[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:08 dist-test-slave-j0qf krb5kdc[5215](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:08 dist-test-slave-j0qf krb5kdc[5215](info): set up 2 sockets
Sep 28 06:35:08 dist-test-slave-j0qf krb5kdc[5215](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 06:35:09.610072 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:35:09 dist-test-slave-j0qf krb5kdc[5215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:35:09.747224 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:44745
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:34319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:44745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:09.859323 5228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:09.859609 5228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:09.859651 5228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:09.863739 5228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:09.863794 5228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:09.863814 5228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:09.863830 5228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:09.863847 5228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:09.869196 5228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:34319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:44745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:44745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:09.870273 5228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:09.871390 5228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:09.880894 5236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:09.880909 5233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:09.880941 5234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:09.881270 5228 server_base.cc:1034] running on GCE node
I20260928 06:35:09.881951 5228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:09.883245 5228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:09.884449 5228 hybrid_clock.cc:648] HybridClock initialized: now 1790577309884419 us; error 43 us; skew 500 ppm
Sep 28 06:35:09 dist-test-slave-j0qf krb5kdc[5215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577309, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:09.892129 5228 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:35:09.893201 5228 webserver.cc:460] Webserver started at http://127.5.22.62:46077/ using document root <none> and password file <none>
I20260928 06:35:09.893815 5228 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:09.893908 5228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:09.894147 5228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:09.896359 5228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "2d8d1dbc7fdf40cc93e34fc934f3f95d"
format_stamp: "Formatted at 2026-09-28 06:35:09 on dist-test-slave-j0qf"
server_key: "cdb78de3f98ef9d4c22797ebe5fc1678"
server_key_iv: "427032893246eb07eceefa4bf455b684"
server_key_version: "encryptionkey@0"
I20260928 06:35:09.896958 5228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "2d8d1dbc7fdf40cc93e34fc934f3f95d"
format_stamp: "Formatted at 2026-09-28 06:35:09 on dist-test-slave-j0qf"
server_key: "cdb78de3f98ef9d4c22797ebe5fc1678"
server_key_iv: "427032893246eb07eceefa4bf455b684"
server_key_version: "encryptionkey@0"
I20260928 06:35:09.901074 5228 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260928 06:35:09.904340 5243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:09.905829 5228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:35:09.906006 5228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "2d8d1dbc7fdf40cc93e34fc934f3f95d"
format_stamp: "Formatted at 2026-09-28 06:35:09 on dist-test-slave-j0qf"
server_key: "cdb78de3f98ef9d4c22797ebe5fc1678"
server_key_iv: "427032893246eb07eceefa4bf455b684"
server_key_version: "encryptionkey@0"
I20260928 06:35:09.906140 5228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:09.936781 5228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:09.940125 5228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:09.940356 5228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:09.948467 5295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:44745 every 8 connection(s)
I20260928 06:35:09.948495 5228 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:44745
I20260928 06:35:09.949705 5228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:09.952733 5296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:09.954682 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 5228
I20260928 06:35:09.954864 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:35:09.956264 5208 external_mini_cluster.cc:1428] Setting key e79da7c9d3a4d3fee80dbdc1cfd63c52
I20260928 06:35:09.958662 5296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d8d1dbc7fdf40cc93e34fc934f3f95d: Bootstrap starting.
I20260928 06:35:09.961551 5296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d8d1dbc7fdf40cc93e34fc934f3f95d: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:09.962371 5296 log.cc:824] T 00000000000000000000000000000000 P 2d8d1dbc7fdf40cc93e34fc934f3f95d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:09.964277 5296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d8d1dbc7fdf40cc93e34fc934f3f95d: No bootstrap required, opened a new log
I20260928 06:35:09.967949 5296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d8d1dbc7fdf40cc93e34fc934f3f95d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8d1dbc7fdf40cc93e34fc934f3f95d" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 44745 } }
I20260928 06:35:09.968194 5296 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d8d1dbc7fdf40cc93e34fc934f3f95d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:09.968276 5296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d8d1dbc7fdf40cc93e34fc934f3f95d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d8d1dbc7fdf40cc93e34fc934f3f95d, State: Initialized, Role: FOLLOWER
I20260928 06:35:09.968734 5296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d8d1dbc7fdf40cc93e34fc934f3f95d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8d1dbc7fdf40cc93e34fc934f3f95d" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 44745 } }
I20260928 06:35:09.968897 5296 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d8d1dbc7fdf40cc93e34fc934f3f95d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:09.968991 5296 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d8d1dbc7fdf40cc93e34fc934f3f95d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:09.969115 5296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d8d1dbc7fdf40cc93e34fc934f3f95d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:09.970037 5296 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d8d1dbc7fdf40cc93e34fc934f3f95d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8d1dbc7fdf40cc93e34fc934f3f95d" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 44745 } }
I20260928 06:35:09.970453 5296 leader_election.cc:302] T 00000000000000000000000000000000 P 2d8d1dbc7fdf40cc93e34fc934f3f95d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d8d1dbc7fdf40cc93e34fc934f3f95d; no voters:
I20260928 06:35:09.970829 5296 leader_election.cc:288] T 00000000000000000000000000000000 P 2d8d1dbc7fdf40cc93e34fc934f3f95d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:09.970937 5301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d8d1dbc7fdf40cc93e34fc934f3f95d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:09.971180 5301 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d8d1dbc7fdf40cc93e34fc934f3f95d [term 1 LEADER]: Becoming Leader. State: Replica: 2d8d1dbc7fdf40cc93e34fc934f3f95d, State: Running, Role: LEADER
I20260928 06:35:09.971534 5301 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d8d1dbc7fdf40cc93e34fc934f3f95d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8d1dbc7fdf40cc93e34fc934f3f95d" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 44745 } }
I20260928 06:35:09.972117 5296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d8d1dbc7fdf40cc93e34fc934f3f95d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:09.972805 5301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d8d1dbc7fdf40cc93e34fc934f3f95d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d8d1dbc7fdf40cc93e34fc934f3f95d. Latest consensus state: current_term: 1 leader_uuid: "2d8d1dbc7fdf40cc93e34fc934f3f95d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8d1dbc7fdf40cc93e34fc934f3f95d" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 44745 } } }
I20260928 06:35:09.972926 5301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d8d1dbc7fdf40cc93e34fc934f3f95d [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:09.973322 5305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:09.974864 5302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d8d1dbc7fdf40cc93e34fc934f3f95d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d8d1dbc7fdf40cc93e34fc934f3f95d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8d1dbc7fdf40cc93e34fc934f3f95d" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 44745 } } }
I20260928 06:35:09.975054 5302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d8d1dbc7fdf40cc93e34fc934f3f95d [sys.catalog]: This master's current role is: LEADER
Sep 28 06:35:09 dist-test-slave-j0qf krb5kdc[5215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:09.976711 5305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:09.981985 5305 catalog_manager.cc:1347] Generated new cluster ID: e60431591cdd4a8baac5577e3fb26b37
I20260928 06:35:09.982050 5305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:09.984493 5298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:09.959730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48606 (local address 127.5.22.62:44745)
0928 06:35:09.960200 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:09.960216 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:09.961310 (+ 1094us) server_negotiation.cc:407] Connection header received
0928 06:35:09.964690 (+ 3380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:09.964707 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:09.965025 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:09.965349 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:09.967770 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:09.969105 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:09.970358 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:09.971087 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:09.972740 (+ 1653us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:09.977150 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:09.977158 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:09.977174 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:09.977206 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:09.979394 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:09.980088 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:09.980095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:09.980100 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:09.980253 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:09.980677 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:09.980680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:09.980682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:09.981196 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:09.981420 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:09.983268 (+ 1848us) server_negotiation.cc:299] Negotiation successful
0928 06:35:09.983557 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":163,"threads_started":1}
I20260928 06:35:09.993196 5305 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:09.994405 5305 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:10.009377 5305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d8d1dbc7fdf40cc93e34fc934f3f95d: Generated new TSK 0
I20260928 06:35:10.010353 5305 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.22.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
I20260928 06:35:10.046602 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:44745
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:34319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:10.161022 5322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:10.161316 5322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:10.161362 5322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:10.165464 5322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:10.165530 5322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:10.165592 5322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:35:10.170845 5322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:34319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:44745
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:10.171828 5322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:10.172955 5322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:10.182466 5330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:10.182459 5328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:10.182459 5327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:10.182632 5322 server_base.cc:1034] running on GCE node
I20260928 06:35:10.183068 5322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:10.183638 5322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:10.184798 5322 hybrid_clock.cc:648] HybridClock initialized: now 1790577310184786 us; error 25 us; skew 500 ppm
Sep 28 06:35:10 dist-test-slave-j0qf krb5kdc[5215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577310, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:10.187549 5322 init.cc:377] Logged in from keytab as kudu/127.5.22.1@KRBTEST.COM (short username kudu)
I20260928 06:35:10.188437 5322 webserver.cc:460] Webserver started at http://127.5.22.1:37117/ using document root <none> and password file <none>
I20260928 06:35:10.188999 5322 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:10.189059 5322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:10.189296 5322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:10.191340 5322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "e2fb59b3728b4ff883261552081d982c"
format_stamp: "Formatted at 2026-09-28 06:35:10 on dist-test-slave-j0qf"
server_key: "150f2befb0d5fc534bd4a5109ff6fa63"
server_key_iv: "639c7cee8d4e6883b033888a4dd28709"
server_key_version: "encryptionkey@0"
I20260928 06:35:10.191891 5322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "e2fb59b3728b4ff883261552081d982c"
format_stamp: "Formatted at 2026-09-28 06:35:10 on dist-test-slave-j0qf"
server_key: "150f2befb0d5fc534bd4a5109ff6fa63"
server_key_iv: "639c7cee8d4e6883b033888a4dd28709"
server_key_version: "encryptionkey@0"
I20260928 06:35:10.195667 5322 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 06:35:10.198040 5337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:10.199471 5322 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:35:10.199637 5322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "e2fb59b3728b4ff883261552081d982c"
format_stamp: "Formatted at 2026-09-28 06:35:10 on dist-test-slave-j0qf"
server_key: "150f2befb0d5fc534bd4a5109ff6fa63"
server_key_iv: "639c7cee8d4e6883b033888a4dd28709"
server_key_version: "encryptionkey@0"
I20260928 06:35:10.199751 5322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:10.221608 5322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:10.224928 5322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:10.225147 5322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:10.225745 5322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:10.226621 5322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:10.226688 5322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:10.226778 5322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:10.226827 5322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:10.235957 5322 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:39165
I20260928 06:35:10.236002 5451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:39165 every 8 connection(s)
I20260928 06:35:10.236884 5322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:10.243173 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 5322
I20260928 06:35:10.243264 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:10.243508 5208 external_mini_cluster.cc:1428] Setting key 3f2501c59affd67961fe8f3ab5dcd049
Sep 28 06:35:10 dist-test-slave-j0qf krb5kdc[5215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577310, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:10.251464 5298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:10.238811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:48739 (local address 127.5.22.62:44745)
0928 06:35:10.238952 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:10.238956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:10.239763 (+ 807us) server_negotiation.cc:407] Connection header received
0928 06:35:10.241180 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:10.241184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:10.241246 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:10.241351 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:10.242684 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:10.243380 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:10.244718 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:10.244976 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:10.245106 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:10.247665 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:10.247673 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:10.247675 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:10.247697 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:10.249326 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:10.249857 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:10.249860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:10.249861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:10.249933 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:10.250283 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:10.250287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:10.250289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:10.250527 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:10.250645 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:10.251144 (+ 499us) server_negotiation.cc:299] Negotiation successful
0928 06:35:10.251285 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:35:10.252293 5454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:10.239171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:44745 (local address 127.5.22.1:48739)
0928 06:35:10.239614 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:10.239659 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:10.240952 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:10.241566 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:10.241573 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:10.241955 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:10.242505 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:10.242521 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:10.243522 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:10.243525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:10.244596 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:10.244605 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:10.245144 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:10.245148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:10.245319 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:10.245962 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:10.245978 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:10.247489 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:10.249444 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:10.249450 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:10.249465 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:10.249734 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:10.250033 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:10.250036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:10.250038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:10.250185 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:10.250655 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:10.250663 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:10.250907 (+ 244us) client_negotiation.cc:769] Sending connection context
0928 06:35:10.251105 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 06:35:10.251429 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.5.22.2@KRBTEST.COM; defaulting to no policy
I20260928 06:35:10.253701 5452 heartbeater.cc:347] Connected to a master server at 127.5.22.62:44745
I20260928 06:35:10.253965 5452 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:10.254588 5452 heartbeater.cc:511] Master 127.5.22.62:44745 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.2@KRBTEST.COM" created.
I20260928 06:35:10.256352 5260 ts_manager.cc:194] Registered new tserver with Master: e2fb59b3728b4ff883261552081d982c (127.5.22.1:39165)
I20260928 06:35:10.258275 5260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.1@KRBTEST.COM'} at 127.5.22.1:48739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
I20260928 06:35:10.273038 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:44745
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:34319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:10.383728 5457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:10.383976 5457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:10.384020 5457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:10.387847 5457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:10.387893 5457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:10.387949 5457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:35:10.393055 5457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:34319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:44745
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:10.393996 5457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:10.394972 5457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:10.404667 5465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:10.404698 5462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:10.404742 5463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:10.404860 5457 server_base.cc:1034] running on GCE node
I20260928 06:35:10.405298 5457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:10.405954 5457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:10.407147 5457 hybrid_clock.cc:648] HybridClock initialized: now 1790577310407116 us; error 49 us; skew 500 ppm
Sep 28 06:35:10 dist-test-slave-j0qf krb5kdc[5215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577310, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:10.409988 5457 init.cc:377] Logged in from keytab as kudu/127.5.22.2@KRBTEST.COM (short username kudu)
I20260928 06:35:10.410974 5457 webserver.cc:460] Webserver started at http://127.5.22.2:39115/ using document root <none> and password file <none>
I20260928 06:35:10.411532 5457 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:10.411631 5457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:10.411865 5457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:10.413842 5457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "287efdea93904ff0aa63a93a6ddf85e5"
format_stamp: "Formatted at 2026-09-28 06:35:10 on dist-test-slave-j0qf"
server_key: "cd71aba65ed3b8ac582f3e08e6324234"
server_key_iv: "92c4eb5849f315f12c72d244bf600f2b"
server_key_version: "encryptionkey@0"
I20260928 06:35:10.414448 5457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "287efdea93904ff0aa63a93a6ddf85e5"
format_stamp: "Formatted at 2026-09-28 06:35:10 on dist-test-slave-j0qf"
server_key: "cd71aba65ed3b8ac582f3e08e6324234"
server_key_iv: "92c4eb5849f315f12c72d244bf600f2b"
server_key_version: "encryptionkey@0"
I20260928 06:35:10.418018 5457 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:10.420361 5473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:10.421599 5457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:10.421727 5457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "287efdea93904ff0aa63a93a6ddf85e5"
format_stamp: "Formatted at 2026-09-28 06:35:10 on dist-test-slave-j0qf"
server_key: "cd71aba65ed3b8ac582f3e08e6324234"
server_key_iv: "92c4eb5849f315f12c72d244bf600f2b"
server_key_version: "encryptionkey@0"
I20260928 06:35:10.421832 5457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:10.432437 5457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:10.435331 5457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:10.435544 5457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:10.436102 5457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:10.436884 5457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:10.436954 5457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:10.437024 5457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:10.437071 5457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:10.446813 5457 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:41741
I20260928 06:35:10.446838 5587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:41741 every 8 connection(s)
I20260928 06:35:10.447757 5457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:10.449591 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 5457
I20260928 06:35:10.449712 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:10.449976 5208 external_mini_cluster.cc:1428] Setting key e75b818c74f9928672051422cc18681e
Sep 28 06:35:10 dist-test-slave-j0qf krb5kdc[5215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577310, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.22.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.3@KRBTEST.COM" created.
I20260928 06:35:10.462929 5298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:10.449621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:55795 (local address 127.5.22.62:44745)
0928 06:35:10.449782 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:10.449787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:10.450904 (+ 1117us) server_negotiation.cc:407] Connection header received
0928 06:35:10.452372 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:10.452376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:10.452443 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:10.452547 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:10.453894 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:10.454597 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:10.455829 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:10.456082 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:10.456212 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:10.459139 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:10.459145 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:10.459149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:10.459184 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:10.460525 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:10.460997 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:10.461000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:10.461001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:10.461076 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:10.461440 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:10.461443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:10.461444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:10.461687 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:10.461794 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:10.462448 (+ 654us) server_negotiation.cc:299] Negotiation successful
0928 06:35:10.462669 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 06:35:10.463611 5590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:10.450165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:44745 (local address 127.5.22.2:55795)
0928 06:35:10.450735 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:10.450791 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:10.452149 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:10.452749 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:10.452756 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:10.453181 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:10.453714 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:10.453728 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:10.454731 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:10.454735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:10.455693 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:10.455703 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:10.456260 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:10.456264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:10.456455 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:10.457115 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:10.457132 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:10.458945 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:10.460630 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:10.460635 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:10.460646 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:10.460898 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:10.461195 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:10.461198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:10.461200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:10.461345 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:10.461787 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:10.461791 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:10.462125 (+ 334us) client_negotiation.cc:769] Sending connection context
0928 06:35:10.462378 (+ 253us) client_negotiation.cc:240] Negotiation successful
0928 06:35:10.462703 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":213,"threads_started":1}
I20260928 06:35:10.464841 5588 heartbeater.cc:347] Connected to a master server at 127.5.22.62:44745
I20260928 06:35:10.465088 5588 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:10.465582 5588 heartbeater.cc:511] Master 127.5.22.62:44745 requested a full tablet report, sending...
I20260928 06:35:10.466658 5260 ts_manager.cc:194] Registered new tserver with Master: 287efdea93904ff0aa63a93a6ddf85e5 (127.5.22.2:41741)
I20260928 06:35:10.467301 5260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.2@KRBTEST.COM'} at 127.5.22.2:55795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
I20260928 06:35:10.483704 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:44745
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:34319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:10.595186 5593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:10.595458 5593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:10.595539 5593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:10.599573 5593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:10.599630 5593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:10.599735 5593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:35:10.605147 5593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:34319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:44745
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:10.606228 5593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:10.607311 5593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:10.617352 5598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:10.617384 5601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:10.617461 5599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:10.617482 5593 server_base.cc:1034] running on GCE node
I20260928 06:35:10.618011 5593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:10.618691 5593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:10.619877 5593 hybrid_clock.cc:648] HybridClock initialized: now 1790577310619844 us; error 51 us; skew 500 ppm
Sep 28 06:35:10 dist-test-slave-j0qf krb5kdc[5215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577310, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:10.622841 5593 init.cc:377] Logged in from keytab as kudu/127.5.22.3@KRBTEST.COM (short username kudu)
I20260928 06:35:10.623713 5593 webserver.cc:460] Webserver started at http://127.5.22.3:42489/ using document root <none> and password file <none>
I20260928 06:35:10.624279 5593 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:10.624367 5593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:10.624646 5593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:10.626817 5593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "74d126bb3f9c463fa8aff206eb1adc47"
format_stamp: "Formatted at 2026-09-28 06:35:10 on dist-test-slave-j0qf"
server_key: "c9128cdb48fd15492aabaa3780fd93f6"
server_key_iv: "5fd5327c7edd7d421262a58599bcd32b"
server_key_version: "encryptionkey@0"
I20260928 06:35:10.627382 5593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "74d126bb3f9c463fa8aff206eb1adc47"
format_stamp: "Formatted at 2026-09-28 06:35:10 on dist-test-slave-j0qf"
server_key: "c9128cdb48fd15492aabaa3780fd93f6"
server_key_iv: "5fd5327c7edd7d421262a58599bcd32b"
server_key_version: "encryptionkey@0"
I20260928 06:35:10.631166 5593 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 06:35:10.633611 5608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:10.634931 5593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 06:35:10.635063 5593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "74d126bb3f9c463fa8aff206eb1adc47"
format_stamp: "Formatted at 2026-09-28 06:35:10 on dist-test-slave-j0qf"
server_key: "c9128cdb48fd15492aabaa3780fd93f6"
server_key_iv: "5fd5327c7edd7d421262a58599bcd32b"
server_key_version: "encryptionkey@0"
I20260928 06:35:10.635177 5593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:10.649183 5593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:10.652740 5593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:10.652988 5593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:10.653687 5593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:10.654603 5593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:10.654675 5593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:10.654747 5593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:10.654788 5593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:10.668526 5593 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:34287
I20260928 06:35:10.668535 5722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:34287 every 8 connection(s)
I20260928 06:35:10.669749 5593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:10.670544 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 5593
I20260928 06:35:10.670660 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:10.670936 5208 external_mini_cluster.cc:1428] Setting key e338a6f162d73f630081801daad7b9dc
Sep 28 06:35:10 dist-test-slave-j0qf krb5kdc[5215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577310, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:10.684248 5298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:10.671945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:51685 (local address 127.5.22.62:44745)
0928 06:35:10.672076 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:10.672081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:10.673246 (+ 1165us) server_negotiation.cc:407] Connection header received
0928 06:35:10.674838 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:10.674841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:10.674887 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:10.674961 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:10.676103 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:10.676684 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:10.678077 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:10.678289 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:10.678444 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:10.680950 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:10.680953 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:10.680955 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:10.680978 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:10.682397 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:10.682891 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:10.682894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:10.682895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:10.682968 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:10.683252 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:10.683254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:10.683255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:10.683414 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:10.683490 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:10.684006 (+ 516us) server_negotiation.cc:299] Negotiation successful
0928 06:35:10.684102 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:35:10.685284 5725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:10.672438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:44745 (local address 127.5.22.3:51685)
0928 06:35:10.673061 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:10.673117 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:10.674634 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:10.675129 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:10.675139 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:10.675503 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:10.675968 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:10.675983 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:10.676821 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:10.676824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:10.677943 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:10.677953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:10.678538 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:10.678541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:10.678702 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:10.679361 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:10.679377 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:10.680802 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:10.682547 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:10.682551 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:10.682562 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:10.682804 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:10.683048 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:10.683050 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:10.683052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:10.683174 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:10.683559 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:10.683564 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:10.683777 (+ 213us) client_negotiation.cc:769] Sending connection context
0928 06:35:10.683933 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 06:35:10.684284 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":175,"threads_started":1}
I20260928 06:35:10.686419 5723 heartbeater.cc:347] Connected to a master server at 127.5.22.62:44745
I20260928 06:35:10.686753 5723 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:10.687410 5723 heartbeater.cc:511] Master 127.5.22.62:44745 requested a full tablet report, sending...
I20260928 06:35:10.688412 5260 ts_manager.cc:194] Registered new tserver with Master: 74d126bb3f9c463fa8aff206eb1adc47 (127.5.22.3:34287)
I20260928 06:35:10.688992 5260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.3@KRBTEST.COM'} at 127.5.22.3:51685
I20260928 06:35:10.697733 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:35:10 dist-test-slave-j0qf krb5kdc[5215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577310, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:35:10 dist-test-slave-j0qf krb5kdc[5215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577310, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.22.1@KRBTEST.COM
I20260928 06:35:10.724555 5730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:10.714729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46090 (local address 127.5.22.1:39165)
0928 06:35:10.715024 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:10.715038 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:10.715085 (+ 47us) server_negotiation.cc:407] Connection header received
0928 06:35:10.715213 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:10.715219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:10.715409 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:10.715525 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:10.716171 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:10.717004 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:10.717868 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:10.718260 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:10.718405 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:10.721058 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:10.721067 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:10.721080 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:10.721104 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:10.722858 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:10.723287 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:10.723292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:10.723293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:10.723369 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:10.723618 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:10.723620 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:10.723621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:10.723854 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:10.724021 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:10.724264 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 06:35:10.724388 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":86,"threads_started":1}
W20260928 06:35:10.725649 5366 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46090
Sep 28 06:35:10 dist-test-slave-j0qf krb5kdc[5215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:35:10 dist-test-slave-j0qf krb5kdc[5215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.1@KRBTEST.COM
I20260928 06:35:10.753146 5730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:10.743890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46102 (local address 127.5.22.1:39165)
0928 06:35:10.744086 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:10.744093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:10.744291 (+ 198us) server_negotiation.cc:407] Connection header received
0928 06:35:10.744470 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:10.744474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:10.744533 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:10.744626 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:10.745356 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:10.745899 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:10.746892 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:10.747154 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:10.747276 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:10.750017 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:10.750020 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:10.750022 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:10.750051 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:10.751435 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:10.751886 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:10.751890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:10.751891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:10.751964 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:10.752254 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:10.752259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:10.752260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:10.752451 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:10.752624 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:10.752843 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 06:35:10.752972 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 06:35:10.754185 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 5322
I20260928 06:35:10.761376 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 5457
I20260928 06:35:10.769009 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 5593
I20260928 06:35:10.774817 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 5228
2026-09-28T06:35:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2287 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:10 dist-test-slave-j0qf krb5kdc[5737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:10 dist-test-slave-j0qf krb5kdc[5737](info): set up 2 sockets
Sep 28 06:35:10 dist-test-slave-j0qf krb5kdc[5737](info): commencing operation
krb5kdc: starting...
W20260928 06:35:11.912806 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:35:11 dist-test-slave-j0qf krb5kdc[5737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:35:12.037768 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:35705
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:43205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:35705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:12.149283 5750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:12.149600 5750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:12.149674 5750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:12.153796 5750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:12.153892 5750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:12.153930 5750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:12.153985 5750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:12.154021 5750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:12.159394 5750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:43205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:35705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:35705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:12.160593 5750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:12.161664 5750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:12.170804 5758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:12.170871 5750 server_base.cc:1034] running on GCE node
W20260928 06:35:12.170809 5755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:12.170804 5756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:12.171634 5750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:12.172781 5750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:12.174012 5750 hybrid_clock.cc:648] HybridClock initialized: now 1790577312173980 us; error 47 us; skew 500 ppm
Sep 28 06:35:12 dist-test-slave-j0qf krb5kdc[5737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577312, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:12.176762 5750 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:35:12.177711 5750 webserver.cc:460] Webserver started at http://127.5.22.62:45495/ using document root <none> and password file <none>
I20260928 06:35:12.178269 5750 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:12.178386 5750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:12.178630 5750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:12.180663 5750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "2ccc7b500cc242539fe1a4c1df08b26b"
format_stamp: "Formatted at 2026-09-28 06:35:12 on dist-test-slave-j0qf"
server_key: "48888c6d6a1b61007c61bdd8686c2bf5"
server_key_iv: "15fa16954a6407fd8a86007d1a300a8e"
server_key_version: "encryptionkey@0"
I20260928 06:35:12.181216 5750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "2ccc7b500cc242539fe1a4c1df08b26b"
format_stamp: "Formatted at 2026-09-28 06:35:12 on dist-test-slave-j0qf"
server_key: "48888c6d6a1b61007c61bdd8686c2bf5"
server_key_iv: "15fa16954a6407fd8a86007d1a300a8e"
server_key_version: "encryptionkey@0"
I20260928 06:35:12.184938 5750 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 06:35:12.187318 5765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:12.188510 5750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:35:12.188642 5750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "2ccc7b500cc242539fe1a4c1df08b26b"
format_stamp: "Formatted at 2026-09-28 06:35:12 on dist-test-slave-j0qf"
server_key: "48888c6d6a1b61007c61bdd8686c2bf5"
server_key_iv: "15fa16954a6407fd8a86007d1a300a8e"
server_key_version: "encryptionkey@0"
I20260928 06:35:12.188748 5750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:12.200013 5750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:12.202765 5750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:12.202956 5750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:12.210165 5750 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:35705
I20260928 06:35:12.210192 5818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:35705 every 8 connection(s)
I20260928 06:35:12.211179 5750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:12.213896 5819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:12.214152 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 5750
I20260928 06:35:12.214277 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:35:12.214576 5208 external_mini_cluster.cc:1428] Setting key 62a2a64740314b2a564b97f2424601df
I20260928 06:35:12.221132 5819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ccc7b500cc242539fe1a4c1df08b26b: Bootstrap starting.
I20260928 06:35:12.223695 5819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ccc7b500cc242539fe1a4c1df08b26b: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:12.224565 5819 log.cc:824] T 00000000000000000000000000000000 P 2ccc7b500cc242539fe1a4c1df08b26b: Log is configured to *not* fsync() on all Append() calls
Sep 28 06:35:12 dist-test-slave-j0qf krb5kdc[5737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:12.226635 5819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ccc7b500cc242539fe1a4c1df08b26b: No bootstrap required, opened a new log
I20260928 06:35:12.230225 5822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:12.215974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50798 (local address 127.5.22.62:35705)
0928 06:35:12.216405 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:12.216423 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:12.216468 (+ 45us) server_negotiation.cc:407] Connection header received
0928 06:35:12.217264 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:12.217288 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:12.217778 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:12.218208 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:12.219051 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:12.220616 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:12.221570 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:12.222239 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:12.222453 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:12.225485 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:12.225492 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:12.225505 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:12.225532 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:12.227374 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:12.227924 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:12.227929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:12.227934 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:12.228049 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:12.228367 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:12.228369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:12.228370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:12.228755 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:12.228905 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:12.229267 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 06:35:12.229498 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":118,"threads_started":1}
I20260928 06:35:12.230969 5819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ccc7b500cc242539fe1a4c1df08b26b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccc7b500cc242539fe1a4c1df08b26b" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } }
I20260928 06:35:12.231227 5819 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ccc7b500cc242539fe1a4c1df08b26b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:12.231312 5819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ccc7b500cc242539fe1a4c1df08b26b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ccc7b500cc242539fe1a4c1df08b26b, State: Initialized, Role: FOLLOWER
I20260928 06:35:12.231810 5819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ccc7b500cc242539fe1a4c1df08b26b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccc7b500cc242539fe1a4c1df08b26b" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } }
I20260928 06:35:12.231978 5819 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ccc7b500cc242539fe1a4c1df08b26b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:12.232070 5819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ccc7b500cc242539fe1a4c1df08b26b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:12.232199 5819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ccc7b500cc242539fe1a4c1df08b26b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:12.233278 5819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ccc7b500cc242539fe1a4c1df08b26b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccc7b500cc242539fe1a4c1df08b26b" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } }
I20260928 06:35:12.233644 5819 leader_election.cc:302] T 00000000000000000000000000000000 P 2ccc7b500cc242539fe1a4c1df08b26b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ccc7b500cc242539fe1a4c1df08b26b; no voters:
I20260928 06:35:12.233999 5819 leader_election.cc:288] T 00000000000000000000000000000000 P 2ccc7b500cc242539fe1a4c1df08b26b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:12.234082 5824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ccc7b500cc242539fe1a4c1df08b26b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:12.234282 5824 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ccc7b500cc242539fe1a4c1df08b26b [term 1 LEADER]: Becoming Leader. State: Replica: 2ccc7b500cc242539fe1a4c1df08b26b, State: Running, Role: LEADER
I20260928 06:35:12.234616 5824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ccc7b500cc242539fe1a4c1df08b26b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccc7b500cc242539fe1a4c1df08b26b" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } }
I20260928 06:35:12.235351 5819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ccc7b500cc242539fe1a4c1df08b26b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:12.236271 5826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ccc7b500cc242539fe1a4c1df08b26b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ccc7b500cc242539fe1a4c1df08b26b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccc7b500cc242539fe1a4c1df08b26b" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } } }
I20260928 06:35:12.236424 5826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ccc7b500cc242539fe1a4c1df08b26b [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:12.236449 5825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ccc7b500cc242539fe1a4c1df08b26b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ccc7b500cc242539fe1a4c1df08b26b. Latest consensus state: current_term: 1 leader_uuid: "2ccc7b500cc242539fe1a4c1df08b26b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccc7b500cc242539fe1a4c1df08b26b" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } } }
I20260928 06:35:12.236559 5825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ccc7b500cc242539fe1a4c1df08b26b [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:12.236845 5833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:12.240173 5833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:12.246281 5833 catalog_manager.cc:1347] Generated new cluster ID: 38cc6d24e8a34c52b7538c1cf66af769
I20260928 06:35:12.246402 5833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:12.259299 5833 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:12.260321 5833 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:12.273085 5833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ccc7b500cc242539fe1a4c1df08b26b: Generated new TSK 0
I20260928 06:35:12.273818 5833 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.22.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
I20260928 06:35:12.311609 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:35705
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:43205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:12.424198 5845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:12.424467 5845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:12.424513 5845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:12.428488 5845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:12.428546 5845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:12.428645 5845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:35:12.433934 5845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:43205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:35705
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:12.435034 5845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:12.436112 5845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:12.445964 5851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:12.446097 5845 server_base.cc:1034] running on GCE node
W20260928 06:35:12.445964 5850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:12.445961 5853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:12.446748 5845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:12.447429 5845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:12.448629 5845 hybrid_clock.cc:648] HybridClock initialized: now 1790577312448601 us; error 33 us; skew 500 ppm
Sep 28 06:35:12 dist-test-slave-j0qf krb5kdc[5737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577312, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:12.451637 5845 init.cc:377] Logged in from keytab as kudu/127.5.22.1@KRBTEST.COM (short username kudu)
I20260928 06:35:12.452441 5845 webserver.cc:460] Webserver started at http://127.5.22.1:46381/ using document root <none> and password file <none>
I20260928 06:35:12.452947 5845 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:12.453002 5845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:12.453169 5845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:12.455663 5845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "bf238434946247acbde1a102ef533813"
format_stamp: "Formatted at 2026-09-28 06:35:12 on dist-test-slave-j0qf"
server_key: "fb4e49ccb8d6fbaf542379b38064270f"
server_key_iv: "68e018ca2200daf659cc06fc22d1e7aa"
server_key_version: "encryptionkey@0"
I20260928 06:35:12.456246 5845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "bf238434946247acbde1a102ef533813"
format_stamp: "Formatted at 2026-09-28 06:35:12 on dist-test-slave-j0qf"
server_key: "fb4e49ccb8d6fbaf542379b38064270f"
server_key_iv: "68e018ca2200daf659cc06fc22d1e7aa"
server_key_version: "encryptionkey@0"
I20260928 06:35:12.460295 5845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:35:12.462772 5860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:12.463975 5845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:35:12.464100 5845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "bf238434946247acbde1a102ef533813"
format_stamp: "Formatted at 2026-09-28 06:35:12 on dist-test-slave-j0qf"
server_key: "fb4e49ccb8d6fbaf542379b38064270f"
server_key_iv: "68e018ca2200daf659cc06fc22d1e7aa"
server_key_version: "encryptionkey@0"
I20260928 06:35:12.464205 5845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:12.480935 5845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:12.484292 5845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:12.484511 5845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:12.485162 5845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:12.485980 5845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:12.486028 5845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:12.486101 5845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:12.486132 5845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:12.495749 5845 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:40461
I20260928 06:35:12.495805 5974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:40461 every 8 connection(s)
I20260928 06:35:12.496678 5845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:12.498561 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 5845
I20260928 06:35:12.498674 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:12.498965 5208 external_mini_cluster.cc:1428] Setting key d16463e692fcd1857e095399aa4e0d25
Sep 28 06:35:12 dist-test-slave-j0qf krb5kdc[5737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577312, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.22.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.2@KRBTEST.COM" created.
I20260928 06:35:12.513170 5822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:12.498709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:40973 (local address 127.5.22.62:35705)
0928 06:35:12.499440 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:12.499444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:12.500006 (+ 562us) server_negotiation.cc:407] Connection header received
0928 06:35:12.501893 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:12.501896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:12.501941 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:12.502017 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:12.503423 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:12.504310 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:12.505703 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:12.506155 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:12.506288 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:12.509164 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:12.509170 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:12.509173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:12.509203 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:12.510794 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:12.511392 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:12.511396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:12.511398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:12.511511 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:12.511935 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:12.511939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:12.511941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:12.512130 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:12.512249 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:12.512855 (+ 606us) server_negotiation.cc:299] Negotiation successful
0928 06:35:12.512994 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621}
I20260928 06:35:12.513967 5977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:12.499251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:35705 (local address 127.5.22.1:40973)
0928 06:35:12.499839 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:12.499896 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:12.501640 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:12.502212 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:12.502224 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:12.502766 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:12.503265 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:12.503280 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:12.504555 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:12.504558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:12.505550 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:12.505560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:12.506370 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:12.506376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:12.506595 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:12.507244 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:12.507260 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:12.508988 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:12.510998 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:12.511004 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:12.511019 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:12.511282 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:12.511616 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:12.511645 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:12.511651 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:12.511813 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:12.512249 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:12.512257 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:12.512619 (+ 362us) client_negotiation.cc:769] Sending connection context
0928 06:35:12.512812 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 06:35:12.513125 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":197,"threads_started":1}
I20260928 06:35:12.515218 5975 heartbeater.cc:347] Connected to a master server at 127.5.22.62:35705
I20260928 06:35:12.515470 5975 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:12.515955 5975 heartbeater.cc:511] Master 127.5.22.62:35705 requested a full tablet report, sending...
I20260928 06:35:12.517585 5782 ts_manager.cc:194] Registered new tserver with Master: bf238434946247acbde1a102ef533813 (127.5.22.1:40461)
I20260928 06:35:12.519304 5782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.1@KRBTEST.COM'} at 127.5.22.1:40973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
I20260928 06:35:12.529570 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:35705
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:43205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:12.640782 5980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:12.641072 5980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:12.641117 5980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:12.645232 5980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:12.645291 5980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:12.645352 5980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:35:12.650729 5980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:43205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:35705
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:12.651738 5980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:12.652779 5980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:12.662818 5985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:12.662835 5986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:12.662806 5980 server_base.cc:1034] running on GCE node
W20260928 06:35:12.662806 5988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:12.663594 5980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:12.664182 5980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:12.665370 5980 hybrid_clock.cc:648] HybridClock initialized: now 1790577312665341 us; error 47 us; skew 500 ppm
Sep 28 06:35:12 dist-test-slave-j0qf krb5kdc[5737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577312, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:12.668310 5980 init.cc:377] Logged in from keytab as kudu/127.5.22.2@KRBTEST.COM (short username kudu)
I20260928 06:35:12.669176 5980 webserver.cc:460] Webserver started at http://127.5.22.2:44361/ using document root <none> and password file <none>
I20260928 06:35:12.669695 5980 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:12.669754 5980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:12.669912 5980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:12.671931 5980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "37105fa5ce9a45bc8e431cfe7b22f3b9"
format_stamp: "Formatted at 2026-09-28 06:35:12 on dist-test-slave-j0qf"
server_key: "40865dadbcda6616e4c1f7dcb1667f5e"
server_key_iv: "fb12b42a5605050634f402593823eeb1"
server_key_version: "encryptionkey@0"
I20260928 06:35:12.672452 5980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "37105fa5ce9a45bc8e431cfe7b22f3b9"
format_stamp: "Formatted at 2026-09-28 06:35:12 on dist-test-slave-j0qf"
server_key: "40865dadbcda6616e4c1f7dcb1667f5e"
server_key_iv: "fb12b42a5605050634f402593823eeb1"
server_key_version: "encryptionkey@0"
I20260928 06:35:12.676091 5980 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 06:35:12.678287 5995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:12.679520 5980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:12.679621 5980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "37105fa5ce9a45bc8e431cfe7b22f3b9"
format_stamp: "Formatted at 2026-09-28 06:35:12 on dist-test-slave-j0qf"
server_key: "40865dadbcda6616e4c1f7dcb1667f5e"
server_key_iv: "fb12b42a5605050634f402593823eeb1"
server_key_version: "encryptionkey@0"
I20260928 06:35:12.679699 5980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:12.695068 5980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:12.698302 5980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:12.698515 5980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:12.699054 5980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:12.699824 5980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:12.699870 5980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:12.699918 5980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:12.699934 5980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:12.708786 5980 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:42601
I20260928 06:35:12.708877 6109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:42601 every 8 connection(s)
I20260928 06:35:12.709782 5980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:12.716032 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 5980
I20260928 06:35:12.716156 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:12.716421 5208 external_mini_cluster.cc:1428] Setting key 6aac778796f04c3cceebddf69b4c5574
Sep 28 06:35:12 dist-test-slave-j0qf krb5kdc[5737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577312, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:12.725212 5822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:12.711782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:53141 (local address 127.5.22.62:35705)
0928 06:35:12.711920 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:12.711924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:12.712895 (+ 971us) server_negotiation.cc:407] Connection header received
0928 06:35:12.714554 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:12.714558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:12.714615 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:12.714728 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:12.716064 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:12.716739 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:12.717993 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:12.718219 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:12.718360 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:12.721110 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:12.721113 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:12.721116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:12.721138 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:12.722691 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:12.723381 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:12.723384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:12.723385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:12.723533 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:12.723992 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:12.723995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:12.723997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:12.724207 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:12.724305 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:12.724974 (+ 669us) server_negotiation.cc:299] Negotiation successful
0928 06:35:12.725083 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:35:12.726521 6112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:12.712144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:35705 (local address 127.5.22.2:53141)
0928 06:35:12.712740 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:12.712785 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:12.714267 (+ 1482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:12.714911 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:12.714919 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:12.715352 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:12.715909 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:12.715924 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:12.716864 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:12.716872 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:12.717858 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:12.717868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:12.718405 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:12.718410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:12.718619 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:12.719262 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:12.719278 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:12.720918 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:12.722812 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:12.722820 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:12.722840 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:12.723262 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:12.723636 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:12.723641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:12.723646 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:12.723848 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:12.724330 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:12.724338 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:12.724686 (+ 348us) client_negotiation.cc:769] Sending connection context
0928 06:35:12.724921 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 06:35:12.725311 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.5.22.3@KRBTEST.COM; defaulting to no policy
I20260928 06:35:12.727871 6110 heartbeater.cc:347] Connected to a master server at 127.5.22.62:35705
I20260928 06:35:12.728087 6110 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:12.728526 6110 heartbeater.cc:511] Master 127.5.22.62:35705 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.3@KRBTEST.COM" created.
I20260928 06:35:12.729859 5782 ts_manager.cc:194] Registered new tserver with Master: 37105fa5ce9a45bc8e431cfe7b22f3b9 (127.5.22.2:42601)
I20260928 06:35:12.730568 5782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.2@KRBTEST.COM'} at 127.5.22.2:53141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
I20260928 06:35:12.749185 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:35705
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:43205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:12.865502 6115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:12.865767 6115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:12.865811 6115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:12.870102 6115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:12.870167 6115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:12.870270 6115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:35:12.875741 6115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:43205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:35705
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:12.876852 6115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:12.877986 6115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:12.888079 6121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:12.888123 6124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:12.888091 6120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:12.888382 6115 server_base.cc:1034] running on GCE node
I20260928 06:35:12.888916 6115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:12.889704 6115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:12.890918 6115 hybrid_clock.cc:648] HybridClock initialized: now 1790577312890898 us; error 32 us; skew 500 ppm
Sep 28 06:35:12 dist-test-slave-j0qf krb5kdc[5737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577312, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:12.894057 6115 init.cc:377] Logged in from keytab as kudu/127.5.22.3@KRBTEST.COM (short username kudu)
I20260928 06:35:12.895004 6115 webserver.cc:460] Webserver started at http://127.5.22.3:41045/ using document root <none> and password file <none>
I20260928 06:35:12.895687 6115 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:12.895783 6115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:12.896023 6115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:12.898145 6115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "ae6c437563714ee885b917afd99a7083"
format_stamp: "Formatted at 2026-09-28 06:35:12 on dist-test-slave-j0qf"
server_key: "7bfb00e1087697aa36908aac7568fcf0"
server_key_iv: "6f2ede9db6615a6db9d7b2629b8cbe06"
server_key_version: "encryptionkey@0"
I20260928 06:35:12.898810 6115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "ae6c437563714ee885b917afd99a7083"
format_stamp: "Formatted at 2026-09-28 06:35:12 on dist-test-slave-j0qf"
server_key: "7bfb00e1087697aa36908aac7568fcf0"
server_key_iv: "6f2ede9db6615a6db9d7b2629b8cbe06"
server_key_version: "encryptionkey@0"
I20260928 06:35:12.902619 6115 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 06:35:12.905092 6131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:12.906581 6115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:12.906718 6115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "ae6c437563714ee885b917afd99a7083"
format_stamp: "Formatted at 2026-09-28 06:35:12 on dist-test-slave-j0qf"
server_key: "7bfb00e1087697aa36908aac7568fcf0"
server_key_iv: "6f2ede9db6615a6db9d7b2629b8cbe06"
server_key_version: "encryptionkey@0"
I20260928 06:35:12.906837 6115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:12.959347 6115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:12.962674 6115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:12.962898 6115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:12.963551 6115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:12.964411 6115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:12.964484 6115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:12.964558 6115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:12.964608 6115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:12.976073 6115 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:43241
I20260928 06:35:12.976131 6245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:43241 every 8 connection(s)
I20260928 06:35:12.977319 6115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:12.986761 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 6115
I20260928 06:35:12.986896 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
Sep 28 06:35:12 dist-test-slave-j0qf krb5kdc[5737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577312, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:12.987210 5208 external_mini_cluster.cc:1428] Setting key 51d12acb225cbd801cbaa0865f42d6da
I20260928 06:35:12.991381 5822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:12.979293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:44945 (local address 127.5.22.62:35705)
0928 06:35:12.979433 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:12.979436 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:12.980208 (+ 772us) server_negotiation.cc:407] Connection header received
0928 06:35:12.981609 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:12.981612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:12.981660 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:12.981739 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:12.983075 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:12.983597 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:12.984771 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:12.984980 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:12.985063 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:12.987824 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:12.987828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:12.987830 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:12.987858 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:12.989465 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:12.989932 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:12.989935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:12.989936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:12.990006 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:12.990443 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:12.990448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:12.990453 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:12.990629 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:12.990708 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:12.991175 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 06:35:12.991266 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:35:12.992292 6249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:12.979594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:35705 (local address 127.5.22.3:44945)
0928 06:35:12.980048 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:12.980093 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:12.981320 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:12.981941 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:12.981948 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:12.982304 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:12.982936 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:12.982951 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:12.983738 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:12.983742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:12.984662 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:12.984672 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:12.985147 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:12.985150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:12.985297 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:12.985909 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:12.985925 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:12.987666 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:12.989574 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:12.989578 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:12.989589 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:12.989850 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:12.990140 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:12.990143 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:12.990144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:12.990286 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:12.990724 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:12.990728 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:12.990963 (+ 235us) client_negotiation.cc:769] Sending connection context
0928 06:35:12.991111 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 06:35:12.991355 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":111,"threads_started":1}
I20260928 06:35:12.993396 6247 heartbeater.cc:347] Connected to a master server at 127.5.22.62:35705
I20260928 06:35:12.993652 6247 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:12.994129 6247 heartbeater.cc:511] Master 127.5.22.62:35705 requested a full tablet report, sending...
I20260928 06:35:12.995170 5782 ts_manager.cc:194] Registered new tserver with Master: ae6c437563714ee885b917afd99a7083 (127.5.22.3:43241)
I20260928 06:35:12.995728 5782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.3@KRBTEST.COM'} at 127.5.22.3:44945
I20260928 06:35:13.001333 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:35:13.012560 5822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:13.004864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50810 (local address 127.5.22.62:35705)
0928 06:35:13.005032 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:13.005036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:13.005207 (+ 171us) server_negotiation.cc:407] Connection header received
0928 06:35:13.005361 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:13.005364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:13.005414 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:13.005497 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:13.006199 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.006809 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.007648 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.007877 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.007984 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:13.009364 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:13.009368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:13.009371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:13.009396 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:13.010799 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:13.011221 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:13.011225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:13.011227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:13.011313 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:13.011595 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:13.011600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:13.011602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:13.011849 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:13.012002 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:13.012254 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 06:35:13.012372 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 06:35:13.017434 5782 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:35:13.020823 5782 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:13.035146 6261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:13.029563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39046 (local address 127.5.22.1:40461)
0928 06:35:13.030107 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:13.030115 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:13.030141 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:35:13.030218 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:13.030223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:13.030447 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:13.030636 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:13.030864 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.032016 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.033551 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.034381 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.034516 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:13.034661 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:13.034855 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 06:35:13.034952 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":75,"threads_started":1}
I20260928 06:35:13.035135 6258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:13.028867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:40461 (local address 127.0.0.1:39046)
0928 06:35:13.029188 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:13.029235 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:13.029354 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:13.030596 (+ 1242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:13.030599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:13.030616 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:13.030725 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:13.030732 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.032137 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.032140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:13.033401 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:13.033411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.034508 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.034512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:13.034659 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:13.034708 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 06:35:13.034823 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 06:35:13.034984 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":53,"thread_start_us":115,"threads_started":1}
I20260928 06:35:13.035574 6259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:13.028965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:42601 (local address 127.0.0.1:49702)
0928 06:35:13.029580 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:13.029594 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:13.029702 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:13.030392 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:13.030401 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:13.030428 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:13.030588 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:13.030595 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.031923 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.031929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:13.034071 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:13.034081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.035178 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.035182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:13.035302 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:13.035336 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 06:35:13.035383 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 06:35:13.035446 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":76,"thread_start_us":116,"threads_started":1}
I20260928 06:35:13.035744 6262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:13.029817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:43241 (local address 127.0.0.1:57120)
0928 06:35:13.030267 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:13.030280 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:13.030402 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:13.031364 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:13.031367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:13.031386 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:13.031465 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:13.031470 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.032656 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.032662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:13.033398 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:13.033407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.034474 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.034478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:13.034648 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:13.034694 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 06:35:13.034811 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 06:35:13.035588 (+ 777us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":693,"thread_start_us":91,"threads_started":1}
I20260928 06:35:13.036021 6263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:13.030043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57120 (local address 127.5.22.3:43241)
0928 06:35:13.030943 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:13.030954 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:13.030974 (+ 20us) server_negotiation.cc:407] Connection header received
0928 06:35:13.031041 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:13.031045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:13.031212 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:13.031357 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:13.031602 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.032519 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.033525 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.034101 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.034227 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:13.034406 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:13.034939 (+ 533us) server_negotiation.cc:299] Negotiation successful
0928 06:35:13.035036 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":792,"thread_start_us":156,"threads_started":1}
I20260928 06:35:13.036348 6260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:13.029388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49702 (local address 127.5.22.2:42601)
0928 06:35:13.029718 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:13.029730 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:13.029756 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:35:13.029830 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:13.029835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:13.030052 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:13.030745 (+ 693us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:13.030905 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.031802 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.034210 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.035029 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.035867 (+ 838us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:13.036018 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:13.036092 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 06:35:13.036204 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":101,"threads_started":1}
I20260928 06:35:13.038122 5908 tablet_service.cc:1467] Processing CreateTablet for tablet ed588dfa74284aa58e8885744335e721 (DEFAULT_TABLE table=test-table [id=3df031b8230d46c69a357ddc2f440d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:13.038729 6172 tablet_service.cc:1467] Processing CreateTablet for tablet ed588dfa74284aa58e8885744335e721 (DEFAULT_TABLE table=test-table [id=3df031b8230d46c69a357ddc2f440d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:13.038727 6043 tablet_service.cc:1467] Processing CreateTablet for tablet ed588dfa74284aa58e8885744335e721 (DEFAULT_TABLE table=test-table [id=3df031b8230d46c69a357ddc2f440d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:13.039273 5908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed588dfa74284aa58e8885744335e721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:13.039741 6043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed588dfa74284aa58e8885744335e721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:13.039741 6172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed588dfa74284aa58e8885744335e721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:13.045765 6264 tablet_bootstrap.cc:492] T ed588dfa74284aa58e8885744335e721 P bf238434946247acbde1a102ef533813: Bootstrap starting.
I20260928 06:35:13.047361 6266 tablet_bootstrap.cc:492] T ed588dfa74284aa58e8885744335e721 P ae6c437563714ee885b917afd99a7083: Bootstrap starting.
I20260928 06:35:13.047699 6265 tablet_bootstrap.cc:492] T ed588dfa74284aa58e8885744335e721 P 37105fa5ce9a45bc8e431cfe7b22f3b9: Bootstrap starting.
I20260928 06:35:13.048086 6264 tablet_bootstrap.cc:654] T ed588dfa74284aa58e8885744335e721 P bf238434946247acbde1a102ef533813: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:13.048942 6264 log.cc:824] T ed588dfa74284aa58e8885744335e721 P bf238434946247acbde1a102ef533813: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:13.049525 6266 tablet_bootstrap.cc:654] T ed588dfa74284aa58e8885744335e721 P ae6c437563714ee885b917afd99a7083: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:13.050199 6265 tablet_bootstrap.cc:654] T ed588dfa74284aa58e8885744335e721 P 37105fa5ce9a45bc8e431cfe7b22f3b9: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:13.050894 6266 log.cc:824] T ed588dfa74284aa58e8885744335e721 P ae6c437563714ee885b917afd99a7083: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:13.050915 6265 log.cc:824] T ed588dfa74284aa58e8885744335e721 P 37105fa5ce9a45bc8e431cfe7b22f3b9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:13.051213 6264 tablet_bootstrap.cc:492] T ed588dfa74284aa58e8885744335e721 P bf238434946247acbde1a102ef533813: No bootstrap required, opened a new log
I20260928 06:35:13.051429 6264 ts_tablet_manager.cc:1397] T ed588dfa74284aa58e8885744335e721 P bf238434946247acbde1a102ef533813: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 06:35:13.052697 6266 tablet_bootstrap.cc:492] T ed588dfa74284aa58e8885744335e721 P ae6c437563714ee885b917afd99a7083: No bootstrap required, opened a new log
I20260928 06:35:13.052908 6266 ts_tablet_manager.cc:1397] T ed588dfa74284aa58e8885744335e721 P ae6c437563714ee885b917afd99a7083: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 06:35:13.052910 6265 tablet_bootstrap.cc:492] T ed588dfa74284aa58e8885744335e721 P 37105fa5ce9a45bc8e431cfe7b22f3b9: No bootstrap required, opened a new log
I20260928 06:35:13.053117 6265 ts_tablet_manager.cc:1397] T ed588dfa74284aa58e8885744335e721 P 37105fa5ce9a45bc8e431cfe7b22f3b9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 06:35:13.055797 6266 raft_consensus.cc:348] T ed588dfa74284aa58e8885744335e721 P ae6c437563714ee885b917afd99a7083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf238434946247acbde1a102ef533813" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40461 } } peers { permanent_uuid: "37105fa5ce9a45bc8e431cfe7b22f3b9" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 42601 } } peers { permanent_uuid: "ae6c437563714ee885b917afd99a7083" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43241 } }
I20260928 06:35:13.056057 6266 raft_consensus.cc:374] T ed588dfa74284aa58e8885744335e721 P ae6c437563714ee885b917afd99a7083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:13.055944 6264 raft_consensus.cc:348] T ed588dfa74284aa58e8885744335e721 P bf238434946247acbde1a102ef533813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf238434946247acbde1a102ef533813" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40461 } } peers { permanent_uuid: "37105fa5ce9a45bc8e431cfe7b22f3b9" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 42601 } } peers { permanent_uuid: "ae6c437563714ee885b917afd99a7083" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43241 } }
I20260928 06:35:13.056128 6266 raft_consensus.cc:729] T ed588dfa74284aa58e8885744335e721 P ae6c437563714ee885b917afd99a7083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae6c437563714ee885b917afd99a7083, State: Initialized, Role: FOLLOWER
I20260928 06:35:13.056192 6264 raft_consensus.cc:374] T ed588dfa74284aa58e8885744335e721 P bf238434946247acbde1a102ef533813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:13.056257 6264 raft_consensus.cc:729] T ed588dfa74284aa58e8885744335e721 P bf238434946247acbde1a102ef533813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf238434946247acbde1a102ef533813, State: Initialized, Role: FOLLOWER
I20260928 06:35:13.056619 6266 consensus_queue.cc:260] T ed588dfa74284aa58e8885744335e721 P ae6c437563714ee885b917afd99a7083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf238434946247acbde1a102ef533813" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40461 } } peers { permanent_uuid: "37105fa5ce9a45bc8e431cfe7b22f3b9" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 42601 } } peers { permanent_uuid: "ae6c437563714ee885b917afd99a7083" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43241 } }
I20260928 06:35:13.056708 6264 consensus_queue.cc:260] T ed588dfa74284aa58e8885744335e721 P bf238434946247acbde1a102ef533813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf238434946247acbde1a102ef533813" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40461 } } peers { permanent_uuid: "37105fa5ce9a45bc8e431cfe7b22f3b9" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 42601 } } peers { permanent_uuid: "ae6c437563714ee885b917afd99a7083" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43241 } }
I20260928 06:35:13.056887 6265 raft_consensus.cc:348] T ed588dfa74284aa58e8885744335e721 P 37105fa5ce9a45bc8e431cfe7b22f3b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf238434946247acbde1a102ef533813" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40461 } } peers { permanent_uuid: "37105fa5ce9a45bc8e431cfe7b22f3b9" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 42601 } } peers { permanent_uuid: "ae6c437563714ee885b917afd99a7083" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43241 } }
I20260928 06:35:13.057330 6247 heartbeater.cc:503] Master 127.5.22.62:35705 was elected leader, sending a full tablet report...
I20260928 06:35:13.057662 6266 ts_tablet_manager.cc:1428] T ed588dfa74284aa58e8885744335e721 P ae6c437563714ee885b917afd99a7083: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 06:35:13.057677 6264 ts_tablet_manager.cc:1428] T ed588dfa74284aa58e8885744335e721 P bf238434946247acbde1a102ef533813: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260928 06:35:13.057840 6265 raft_consensus.cc:374] T ed588dfa74284aa58e8885744335e721 P 37105fa5ce9a45bc8e431cfe7b22f3b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:13.057904 6265 raft_consensus.cc:729] T ed588dfa74284aa58e8885744335e721 P 37105fa5ce9a45bc8e431cfe7b22f3b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37105fa5ce9a45bc8e431cfe7b22f3b9, State: Initialized, Role: FOLLOWER
I20260928 06:35:13.057987 5975 heartbeater.cc:503] Master 127.5.22.62:35705 was elected leader, sending a full tablet report...
I20260928 06:35:13.058368 6265 consensus_queue.cc:260] T ed588dfa74284aa58e8885744335e721 P 37105fa5ce9a45bc8e431cfe7b22f3b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf238434946247acbde1a102ef533813" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40461 } } peers { permanent_uuid: "37105fa5ce9a45bc8e431cfe7b22f3b9" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 42601 } } peers { permanent_uuid: "ae6c437563714ee885b917afd99a7083" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43241 } }
I20260928 06:35:13.059245 6265 ts_tablet_manager.cc:1428] T ed588dfa74284aa58e8885744335e721 P 37105fa5ce9a45bc8e431cfe7b22f3b9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 06:35:13.059268 6110 heartbeater.cc:503] Master 127.5.22.62:35705 was elected leader, sending a full tablet report...
I20260928 06:35:13.076311 6272 raft_consensus.cc:482] T ed588dfa74284aa58e8885744335e721 P 37105fa5ce9a45bc8e431cfe7b22f3b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:13.076561 6272 raft_consensus.cc:504] T ed588dfa74284aa58e8885744335e721 P 37105fa5ce9a45bc8e431cfe7b22f3b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf238434946247acbde1a102ef533813" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40461 } } peers { permanent_uuid: "37105fa5ce9a45bc8e431cfe7b22f3b9" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 42601 } } peers { permanent_uuid: "ae6c437563714ee885b917afd99a7083" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43241 } }
I20260928 06:35:13.077708 6272 leader_election.cc:288] T ed588dfa74284aa58e8885744335e721 P 37105fa5ce9a45bc8e431cfe7b22f3b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf238434946247acbde1a102ef533813 (127.5.22.1:40461), ae6c437563714ee885b917afd99a7083 (127.5.22.3:43241)
I20260928 06:35:13.082716 6263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:13.078056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:58703 (local address 127.5.22.3:43241)
0928 06:35:13.078180 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:13.078184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:13.078492 (+ 308us) server_negotiation.cc:407] Connection header received
0928 06:35:13.078627 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:13.078631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:13.078674 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:13.078739 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:13.079008 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.079719 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.081258 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.081884 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.082011 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:13.082085 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:13.082511 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 06:35:13.082593 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 06:35:13.082751 6261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:13.077976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:60797 (local address 127.5.22.1:40461)
0928 06:35:13.078109 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:13.078115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:13.078162 (+ 47us) server_negotiation.cc:407] Connection header received
0928 06:35:13.078359 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:13.078364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:13.078418 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:13.078509 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:13.078830 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.079628 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.081258 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.081902 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.081999 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:13.082065 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:13.082577 (+ 512us) server_negotiation.cc:299] Negotiation successful
0928 06:35:13.082651 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 06:35:13.082733 6112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:13.077909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:40461 (local address 127.5.22.2:60797)
0928 06:35:13.078092 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:13.078108 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:13.078207 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:13.078545 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:13.078551 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:13.078570 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:13.078692 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:13.078697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.079758 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.079761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:13.081114 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:13.081125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.082023 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.082027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:13.082331 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:13.082389 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 06:35:13.082463 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 06:35:13.082580 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":173}
I20260928 06:35:13.082742 6273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:13.077967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:43241 (local address 127.5.22.2:58703)
0928 06:35:13.078396 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:13.078408 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:13.078518 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:13.078780 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:13.078785 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:13.078804 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:13.078900 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:13.078904 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.079836 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.079839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:13.081114 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:13.081125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.082023 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.082027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:13.082327 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:13.082389 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 06:35:13.082463 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 06:35:13.082580 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":145,"thread_start_us":149,"threads_started":1}
I20260928 06:35:13.083488 6200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed588dfa74284aa58e8885744335e721" candidate_uuid: "37105fa5ce9a45bc8e431cfe7b22f3b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae6c437563714ee885b917afd99a7083" is_pre_election: true
I20260928 06:35:13.083509 5929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed588dfa74284aa58e8885744335e721" candidate_uuid: "37105fa5ce9a45bc8e431cfe7b22f3b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf238434946247acbde1a102ef533813" is_pre_election: true
I20260928 06:35:13.083822 5929 raft_consensus.cc:2454] T ed588dfa74284aa58e8885744335e721 P bf238434946247acbde1a102ef533813 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37105fa5ce9a45bc8e431cfe7b22f3b9 in term 0.
I20260928 06:35:13.083828 6200 raft_consensus.cc:2454] T ed588dfa74284aa58e8885744335e721 P ae6c437563714ee885b917afd99a7083 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37105fa5ce9a45bc8e431cfe7b22f3b9 in term 0.
I20260928 06:35:13.084329 5996 leader_election.cc:302] T ed588dfa74284aa58e8885744335e721 P 37105fa5ce9a45bc8e431cfe7b22f3b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 37105fa5ce9a45bc8e431cfe7b22f3b9, ae6c437563714ee885b917afd99a7083, bf238434946247acbde1a102ef533813; no voters:
I20260928 06:35:13.084594 6272 raft_consensus.cc:2781] T ed588dfa74284aa58e8885744335e721 P 37105fa5ce9a45bc8e431cfe7b22f3b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:13.084689 6272 raft_consensus.cc:482] T ed588dfa74284aa58e8885744335e721 P 37105fa5ce9a45bc8e431cfe7b22f3b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:13.084748 6272 raft_consensus.cc:3037] T ed588dfa74284aa58e8885744335e721 P 37105fa5ce9a45bc8e431cfe7b22f3b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:13.085752 6272 raft_consensus.cc:504] T ed588dfa74284aa58e8885744335e721 P 37105fa5ce9a45bc8e431cfe7b22f3b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf238434946247acbde1a102ef533813" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40461 } } peers { permanent_uuid: "37105fa5ce9a45bc8e431cfe7b22f3b9" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 42601 } } peers { permanent_uuid: "ae6c437563714ee885b917afd99a7083" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43241 } }
I20260928 06:35:13.086120 6272 leader_election.cc:288] T ed588dfa74284aa58e8885744335e721 P 37105fa5ce9a45bc8e431cfe7b22f3b9 [CANDIDATE]: Term 1 election: Requested vote from peers bf238434946247acbde1a102ef533813 (127.5.22.1:40461), ae6c437563714ee885b917afd99a7083 (127.5.22.3:43241)
I20260928 06:35:13.086565 5929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed588dfa74284aa58e8885744335e721" candidate_uuid: "37105fa5ce9a45bc8e431cfe7b22f3b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf238434946247acbde1a102ef533813"
I20260928 06:35:13.086565 6200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed588dfa74284aa58e8885744335e721" candidate_uuid: "37105fa5ce9a45bc8e431cfe7b22f3b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae6c437563714ee885b917afd99a7083"
I20260928 06:35:13.086706 5929 raft_consensus.cc:3037] T ed588dfa74284aa58e8885744335e721 P bf238434946247acbde1a102ef533813 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:13.086706 6200 raft_consensus.cc:3037] T ed588dfa74284aa58e8885744335e721 P ae6c437563714ee885b917afd99a7083 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:13.088140 5929 raft_consensus.cc:2454] T ed588dfa74284aa58e8885744335e721 P bf238434946247acbde1a102ef533813 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37105fa5ce9a45bc8e431cfe7b22f3b9 in term 1.
I20260928 06:35:13.088140 6200 raft_consensus.cc:2454] T ed588dfa74284aa58e8885744335e721 P ae6c437563714ee885b917afd99a7083 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37105fa5ce9a45bc8e431cfe7b22f3b9 in term 1.
I20260928 06:35:13.088531 5997 leader_election.cc:302] T ed588dfa74284aa58e8885744335e721 P 37105fa5ce9a45bc8e431cfe7b22f3b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37105fa5ce9a45bc8e431cfe7b22f3b9, ae6c437563714ee885b917afd99a7083; no voters:
I20260928 06:35:13.088745 6272 raft_consensus.cc:2781] T ed588dfa74284aa58e8885744335e721 P 37105fa5ce9a45bc8e431cfe7b22f3b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:13.089037 6272 raft_consensus.cc:686] T ed588dfa74284aa58e8885744335e721 P 37105fa5ce9a45bc8e431cfe7b22f3b9 [term 1 LEADER]: Becoming Leader. State: Replica: 37105fa5ce9a45bc8e431cfe7b22f3b9, State: Running, Role: LEADER
I20260928 06:35:13.089481 6272 consensus_queue.cc:237] T ed588dfa74284aa58e8885744335e721 P 37105fa5ce9a45bc8e431cfe7b22f3b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf238434946247acbde1a102ef533813" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40461 } } peers { permanent_uuid: "37105fa5ce9a45bc8e431cfe7b22f3b9" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 42601 } } peers { permanent_uuid: "ae6c437563714ee885b917afd99a7083" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43241 } }
I20260928 06:35:13.093091 5782 catalog_manager.cc:5495] T ed588dfa74284aa58e8885744335e721 P 37105fa5ce9a45bc8e431cfe7b22f3b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37105fa5ce9a45bc8e431cfe7b22f3b9 (127.5.22.2). New cstate: current_term: 1 leader_uuid: "37105fa5ce9a45bc8e431cfe7b22f3b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf238434946247acbde1a102ef533813" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37105fa5ce9a45bc8e431cfe7b22f3b9" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 42601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae6c437563714ee885b917afd99a7083" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43241 } health_report { overall_health: UNKNOWN } } }
Sep 28 06:35:13 dist-test-slave-j0qf krb5kdc[5737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.1@KRBTEST.COM
I20260928 06:35:13.115800 6261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:13.106381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39052 (local address 127.5.22.1:40461)
0928 06:35:13.106539 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:13.106543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:13.106724 (+ 181us) server_negotiation.cc:407] Connection header received
0928 06:35:13.106830 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:13.106832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:13.106875 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:13.106944 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:13.107612 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.108135 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.109031 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.109250 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.109338 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:13.111889 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:13.111901 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:13.111915 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:13.111947 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:13.114047 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:13.114540 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:13.114547 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:13.114548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:13.114636 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:13.114864 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:13.114866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:13.114867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:13.115133 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:13.115294 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:13.115452 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 06:35:13.115559 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 06:35:13 dist-test-slave-j0qf krb5kdc[5737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577313, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:35:13 dist-test-slave-j0qf krb5kdc[5737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577313, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.22.1@KRBTEST.COM
I20260928 06:35:13.145339 6261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:13.136483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39054 (local address 127.5.22.1:40461)
0928 06:35:13.136697 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:13.136702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:13.136830 (+ 128us) server_negotiation.cc:407] Connection header received
0928 06:35:13.137014 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:13.137018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:13.137073 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:13.137172 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:13.137928 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.138548 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.139392 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.139647 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.139761 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:13.142262 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:13.142265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:13.142267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:13.142290 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:13.143677 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:13.144109 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:13.144112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:13.144113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:13.144187 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:13.144554 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:13.144557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:13.144559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:13.144721 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:13.144821 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:13.145114 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 06:35:13.145217 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
Sep 28 06:35:13 dist-test-slave-j0qf krb5kdc[5737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:35:13 dist-test-slave-j0qf krb5kdc[5737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.1@KRBTEST.COM
I20260928 06:35:13.171530 6261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:13.162861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39058 (local address 127.5.22.1:40461)
0928 06:35:13.163049 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:13.163053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:13.163133 (+ 80us) server_negotiation.cc:407] Connection header received
0928 06:35:13.163299 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:13.163303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:13.163349 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:13.163428 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:13.164096 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.164598 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.165413 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:13.165752 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:13.165854 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:13.168558 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:13.168562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:13.168564 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:13.168588 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:13.170066 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:13.170517 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:13.170519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:13.170520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:13.170597 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:13.170805 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:13.170807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:13.170808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:13.170978 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:13.171074 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:13.171243 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 06:35:13.171337 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260928 06:35:13.172991 5888 tablet_service.cc:2908] Rejecting scan request for tablet ed588dfa74284aa58e8885744335e721: Uninitialized: clean time has not yet been initialized
I20260928 06:35:13.175148 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 5845
I20260928 06:35:13.181779 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 5980
I20260928 06:35:13.188059 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 6115
I20260928 06:35:13.193969 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 5750
2026-09-28T06:35:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2419 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:13 dist-test-slave-j0qf krb5kdc[6288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:13 dist-test-slave-j0qf krb5kdc[6288](info): set up 2 sockets
Sep 28 06:35:13 dist-test-slave-j0qf krb5kdc[6288](info): commencing operation
krb5kdc: starting...
W20260928 06:35:14.312608 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:35:14 dist-test-slave-j0qf krb5kdc[6288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:35:14.441777 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:33127
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:37869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:33127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:14.565608 6301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:14.565872 6301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:14.565950 6301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:14.569972 6301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:14.570039 6301 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 06:35:14.570055 6301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:14.570073 6301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:14.570089 6301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:14.570149 6301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:14.575577 6301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:37869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:33127
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:33127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:14.576668 6301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:14.577768 6301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:14.586313 6307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:14.586336 6306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:14.586344 6309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:14.586390 6301 server_base.cc:1034] running on GCE node
I20260928 06:35:14.587282 6301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:14.588428 6301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:14.589623 6301 hybrid_clock.cc:648] HybridClock initialized: now 1790577314589604 us; error 31 us; skew 500 ppm
Sep 28 06:35:14 dist-test-slave-j0qf krb5kdc[6288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577314, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:14.592449 6301 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:35:14.593339 6301 webserver.cc:460] Webserver started at http://127.5.22.62:39991/ using document root <none> and password file <none>
I20260928 06:35:14.593890 6301 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:14.593999 6301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:14.594233 6301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:14.596218 6301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "179eaae02aec40a48344f40bf270f9d8"
format_stamp: "Formatted at 2026-09-28 06:35:14 on dist-test-slave-j0qf"
server_key: "e3df3a21380359fede92e48c581e9ea0"
server_key_iv: "715c496ee65e19b45f81be16deb29e5f"
server_key_version: "encryptionkey@0"
I20260928 06:35:14.596768 6301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "179eaae02aec40a48344f40bf270f9d8"
format_stamp: "Formatted at 2026-09-28 06:35:14 on dist-test-slave-j0qf"
server_key: "e3df3a21380359fede92e48c581e9ea0"
server_key_iv: "715c496ee65e19b45f81be16deb29e5f"
server_key_version: "encryptionkey@0"
I20260928 06:35:14.600368 6301 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 06:35:14.602789 6316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:14.604313 6301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:14.604576 6301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "179eaae02aec40a48344f40bf270f9d8"
format_stamp: "Formatted at 2026-09-28 06:35:14 on dist-test-slave-j0qf"
server_key: "e3df3a21380359fede92e48c581e9ea0"
server_key_iv: "715c496ee65e19b45f81be16deb29e5f"
server_key_version: "encryptionkey@0"
I20260928 06:35:14.604727 6301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:14.634558 6301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:14.637789 6301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:14.638021 6301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:14.646675 6301 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:33127
I20260928 06:35:14.646708 6378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:33127 every 8 connection(s)
I20260928 06:35:14.647701 6301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:14.648470 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 6301
I20260928 06:35:14.648597 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:35:14.648870 5208 external_mini_cluster.cc:1428] Setting key c9f5100b122973d4f4b8cea67234b48a
I20260928 06:35:14.651185 6379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:14.657100 6379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 179eaae02aec40a48344f40bf270f9d8: Bootstrap starting.
Sep 28 06:35:14 dist-test-slave-j0qf krb5kdc[6288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:14.659513 6379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 179eaae02aec40a48344f40bf270f9d8: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:14.660705 6379 log.cc:824] T 00000000000000000000000000000000 P 179eaae02aec40a48344f40bf270f9d8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:14.663101 6379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 179eaae02aec40a48344f40bf270f9d8: No bootstrap required, opened a new log
I20260928 06:35:14.664367 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:14.650493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46236 (local address 127.5.22.62:33127)
0928 06:35:14.650936 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:14.650952 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:14.650992 (+ 40us) server_negotiation.cc:407] Connection header received
0928 06:35:14.651700 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:14.651718 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:14.652051 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:14.652419 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:14.653653 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:14.654770 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:14.655638 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:14.656123 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:14.656295 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:14.659113 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:14.659118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:14.659132 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:14.659161 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:14.661111 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:14.661644 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:14.661650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:14.661656 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:14.661793 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:14.662078 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:14.662081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:14.662083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:14.662679 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:14.662870 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:14.663139 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 06:35:14.663464 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":146,"threads_started":1}
I20260928 06:35:14.666716 6379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 179eaae02aec40a48344f40bf270f9d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179eaae02aec40a48344f40bf270f9d8" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 33127 } }
I20260928 06:35:14.666929 6379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 179eaae02aec40a48344f40bf270f9d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:14.666965 6379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 179eaae02aec40a48344f40bf270f9d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 179eaae02aec40a48344f40bf270f9d8, State: Initialized, Role: FOLLOWER
I20260928 06:35:14.667351 6379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 179eaae02aec40a48344f40bf270f9d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179eaae02aec40a48344f40bf270f9d8" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 33127 } }
I20260928 06:35:14.667472 6379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 179eaae02aec40a48344f40bf270f9d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:14.667512 6379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 179eaae02aec40a48344f40bf270f9d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:14.667573 6379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 179eaae02aec40a48344f40bf270f9d8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:14.668557 6379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 179eaae02aec40a48344f40bf270f9d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179eaae02aec40a48344f40bf270f9d8" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 33127 } }
I20260928 06:35:14.668856 6379 leader_election.cc:302] T 00000000000000000000000000000000 P 179eaae02aec40a48344f40bf270f9d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 179eaae02aec40a48344f40bf270f9d8; no voters:
I20260928 06:35:14.669114 6379 leader_election.cc:288] T 00000000000000000000000000000000 P 179eaae02aec40a48344f40bf270f9d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:14.669256 6385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 179eaae02aec40a48344f40bf270f9d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:14.669476 6385 raft_consensus.cc:686] T 00000000000000000000000000000000 P 179eaae02aec40a48344f40bf270f9d8 [term 1 LEADER]: Becoming Leader. State: Replica: 179eaae02aec40a48344f40bf270f9d8, State: Running, Role: LEADER
I20260928 06:35:14.669817 6385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 179eaae02aec40a48344f40bf270f9d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179eaae02aec40a48344f40bf270f9d8" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 33127 } }
I20260928 06:35:14.670307 6379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 179eaae02aec40a48344f40bf270f9d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:14.671638 6387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 179eaae02aec40a48344f40bf270f9d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 179eaae02aec40a48344f40bf270f9d8. Latest consensus state: current_term: 1 leader_uuid: "179eaae02aec40a48344f40bf270f9d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179eaae02aec40a48344f40bf270f9d8" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 33127 } } }
I20260928 06:35:14.671646 6386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 179eaae02aec40a48344f40bf270f9d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "179eaae02aec40a48344f40bf270f9d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179eaae02aec40a48344f40bf270f9d8" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 33127 } } }
I20260928 06:35:14.671764 6387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 179eaae02aec40a48344f40bf270f9d8 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:14.671767 6386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 179eaae02aec40a48344f40bf270f9d8 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:14.672096 6394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:14.675139 6394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:14.680202 6394 catalog_manager.cc:1347] Generated new cluster ID: 4428cb13eff947268d25e72fbb7f35c5
I20260928 06:35:14.680269 6394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:14.699150 6394 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:14.700074 6394 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:14.709579 6394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 179eaae02aec40a48344f40bf270f9d8: Generated new TSK 0
I20260928 06:35:14.710462 6394 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.22.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
I20260928 06:35:14.746084 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:33127
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:37869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:14.860000 6406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:14.860251 6406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:14.860293 6406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:14.864244 6406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:14.864297 6406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:14.864334 6406 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:35:14.864365 6406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:35:14.869326 6406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:37869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.22.62:33127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:14.870239 6406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:14.871364 6406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:14.880998 6411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:14.881019 6414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:14.880998 6412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:14.881342 6406 server_base.cc:1034] running on GCE node
I20260928 06:35:14.881762 6406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:14.882475 6406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:14.883641 6406 hybrid_clock.cc:648] HybridClock initialized: now 1790577314883624 us; error 33 us; skew 500 ppm
Sep 28 06:35:14 dist-test-slave-j0qf krb5kdc[6288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577314, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:14.886544 6406 init.cc:377] Logged in from keytab as kudu/127.5.22.1@KRBTEST.COM (short username kudu)
I20260928 06:35:14.887423 6406 webserver.cc:460] Webserver started at http://127.5.22.1:42207/ using document root <none> and password file <none>
I20260928 06:35:14.887984 6406 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:14.888075 6406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:14.888309 6406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:14.890648 6406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "5241ea335d87483b85f2a0e0c9d727ca"
format_stamp: "Formatted at 2026-09-28 06:35:14 on dist-test-slave-j0qf"
server_key: "d6972441af0822c6d53d5af02ca434e2"
server_key_iv: "bd5794431bd0b741a79dd074a134bfaf"
server_key_version: "encryptionkey@0"
I20260928 06:35:14.891234 6406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "5241ea335d87483b85f2a0e0c9d727ca"
format_stamp: "Formatted at 2026-09-28 06:35:14 on dist-test-slave-j0qf"
server_key: "d6972441af0822c6d53d5af02ca434e2"
server_key_iv: "bd5794431bd0b741a79dd074a134bfaf"
server_key_version: "encryptionkey@0"
I20260928 06:35:14.895175 6406 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 06:35:14.897603 6421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:14.899008 6406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:35:14.899164 6406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "5241ea335d87483b85f2a0e0c9d727ca"
format_stamp: "Formatted at 2026-09-28 06:35:14 on dist-test-slave-j0qf"
server_key: "d6972441af0822c6d53d5af02ca434e2"
server_key_iv: "bd5794431bd0b741a79dd074a134bfaf"
server_key_version: "encryptionkey@0"
I20260928 06:35:14.899288 6406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:14.917476 6406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:14.920956 6406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:14.921176 6406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:14.922835 6406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:14.922910 6406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:14.922986 6406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:14.923030 6406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:35:14 dist-test-slave-j0qf krb5kdc[6288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577314, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:14.934352 6406 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:39565
I20260928 06:35:14.934466 6537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:39565 every 8 connection(s)
I20260928 06:35:14.935477 6406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:14.937398 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:14.923902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:37185 (local address 127.5.22.62:33127)
0928 06:35:14.924065 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:14.924070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:14.924924 (+ 854us) server_negotiation.cc:407] Connection header received
0928 06:35:14.926089 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:14.926096 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:14.926168 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:14.926280 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:14.927658 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:14.928213 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:14.929362 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:14.929614 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:14.929756 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:14.933010 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:14.933015 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:14.933018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:14.933057 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:14.934974 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:14.935637 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:14.935642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:14.935643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:14.935745 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:14.936179 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:14.936185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:14.936188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:14.936448 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:14.936577 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:14.937078 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 06:35:14.937220 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 06:35:14.938213 6430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:14.924294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:33127 (local address 127.5.22.1:37185)
0928 06:35:14.924756 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:14.924807 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:14.925830 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:14.926539 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:14.926546 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:14.927007 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:14.927492 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:14.927505 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:14.928343 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:14.928346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:14.929231 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:14.929240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:14.929799 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:14.929804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:14.930006 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:14.930725 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:14.930748 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:14.932759 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:14.935096 (+ 2337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:14.935105 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:14.935123 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:14.935466 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:14.935853 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:14.935859 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:14.935863 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:14.936053 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:14.936567 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:14.936574 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:14.936835 (+ 261us) client_negotiation.cc:769] Sending connection context
0928 06:35:14.937032 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 06:35:14.937369 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":142,"threads_started":1}
I20260928 06:35:14.939584 6538 heartbeater.cc:347] Connected to a master server at 127.5.22.62:33127
I20260928 06:35:14.939738 6538 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:14.940369 6538 heartbeater.cc:511] Master 127.5.22.62:33127 requested a full tablet report, sending...
I20260928 06:35:14.942193 6333 ts_manager.cc:194] Registered new tserver with Master: 5241ea335d87483b85f2a0e0c9d727ca (127.5.22.1:39565)
I20260928 06:35:14.943187 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 6406
I20260928 06:35:14.943334 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:14.943605 6333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.1@KRBTEST.COM'} at 127.5.22.1:37185
I20260928 06:35:14.943647 5208 external_mini_cluster.cc:1428] Setting key fcbd0e6b852208ecff1770da068e1ec8
I20260928 06:35:14.949874 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:14.942173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:44285 (local address 127.5.22.62:33127)
0928 06:35:14.942308 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:14.942351 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:14.942627 (+ 276us) server_negotiation.cc:407] Connection header received
0928 06:35:14.942834 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:14.942838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:14.942916 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:14.943006 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:14.943494 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:14.944262 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:14.945287 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:14.945515 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:14.945621 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:14.946821 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:14.946824 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:14.946829 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:14.946854 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:14.948292 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:14.948717 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:14.948728 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:14.948729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:14.948801 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:14.949082 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:14.949087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:14.949089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:14.949253 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:14.949360 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:14.949615 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 06:35:14.949720 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 06:35:14.949894 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:14.942074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:33127 (local address 127.5.22.1:44285)
0928 06:35:14.942531 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:14.942545 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:14.942683 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:14.943059 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:14.943062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:14.943269 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:14.943384 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:14.943390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:14.944399 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:14.944404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:14.945160 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:14.945170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:14.945637 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:14.945640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:14.945735 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:14.946280 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:14.946293 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:14.946679 (+ 386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:14.948407 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:14.948412 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:14.948414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:14.948613 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:14.948893 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:14.948897 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:14.948899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:14.948991 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:14.949350 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:14.949353 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:14.949450 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 06:35:14.949560 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 06:35:14.949706 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":3,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.5.22.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
I20260928 06:35:14.973766 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:33127
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:37869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:15.084256 6547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:15.084509 6547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:15.084554 6547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:15.088593 6547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:15.088649 6547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:15.088687 6547 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:35:15.088752 6547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:35:15.094002 6547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:37869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.22.62:33127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:15.095100 6547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:15.096128 6547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:15.106115 6555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:15.106161 6553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:15.106163 6547 server_base.cc:1034] running on GCE node
W20260928 06:35:15.106182 6552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:15.106803 6547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:15.107600 6547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:15.108796 6547 hybrid_clock.cc:648] HybridClock initialized: now 1790577315108776 us; error 33 us; skew 500 ppm
Sep 28 06:35:15 dist-test-slave-j0qf krb5kdc[6288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577315, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:15.111907 6547 init.cc:377] Logged in from keytab as kudu/127.5.22.2@KRBTEST.COM (short username kudu)
I20260928 06:35:15.112859 6547 webserver.cc:460] Webserver started at http://127.5.22.2:39447/ using document root <none> and password file <none>
I20260928 06:35:15.113436 6547 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:15.113529 6547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:15.113821 6547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:15.115967 6547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "4656c42c39664c43be519e461ac32ac0"
format_stamp: "Formatted at 2026-09-28 06:35:15 on dist-test-slave-j0qf"
server_key: "0e4ff2db7c6a7c1dc830755c75a151b7"
server_key_iv: "e58501aeed1c396817ac4d25c3e633d9"
server_key_version: "encryptionkey@0"
I20260928 06:35:15.116586 6547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "4656c42c39664c43be519e461ac32ac0"
format_stamp: "Formatted at 2026-09-28 06:35:15 on dist-test-slave-j0qf"
server_key: "0e4ff2db7c6a7c1dc830755c75a151b7"
server_key_iv: "e58501aeed1c396817ac4d25c3e633d9"
server_key_version: "encryptionkey@0"
I20260928 06:35:15.120433 6547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:35:15.122937 6562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:15.124321 6547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:15.124457 6547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "4656c42c39664c43be519e461ac32ac0"
format_stamp: "Formatted at 2026-09-28 06:35:15 on dist-test-slave-j0qf"
server_key: "0e4ff2db7c6a7c1dc830755c75a151b7"
server_key_iv: "e58501aeed1c396817ac4d25c3e633d9"
server_key_version: "encryptionkey@0"
I20260928 06:35:15.124574 6547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:15.148102 6547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:15.151649 6547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:15.151862 6547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:15.153414 6547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:15.153496 6547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:15.153584 6547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:15.153625 6547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:35:15 dist-test-slave-j0qf krb5kdc[6288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577315, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:15.164906 6547 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:36693
I20260928 06:35:15.165006 6677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:36693 every 8 connection(s)
I20260928 06:35:15.165941 6547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:15.168033 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:15.154404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:45599 (local address 127.5.22.62:33127)
0928 06:35:15.154531 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:15.154542 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:15.155659 (+ 1117us) server_negotiation.cc:407] Connection header received
0928 06:35:15.156782 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:15.156787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:15.156858 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:15.156972 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:15.158577 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.159110 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.160277 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.160592 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.160710 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:15.164081 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:15.164084 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:15.164086 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:15.164113 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:15.165726 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:15.166318 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:15.166355 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:15.166359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:15.166470 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:15.166944 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:15.166948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:15.166950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:15.167192 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:15.167277 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:15.167779 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 06:35:15.167913 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:35:15.168816 6571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:15.154974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:33127 (local address 127.5.22.2:45599)
0928 06:35:15.155469 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:15.155514 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:15.156476 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:15.157212 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:15.157227 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:15.157684 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:15.158419 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:15.158436 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.159235 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.159239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:15.160142 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:15.160151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.160764 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.160768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:15.160967 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:15.161581 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:15.161599 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:15.163888 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:15.165842 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:15.165850 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:15.165867 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:15.166195 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:15.166638 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:15.166641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:15.166643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:15.166808 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:15.167288 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:15.167292 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:15.167517 (+ 225us) client_negotiation.cc:769] Sending connection context
0928 06:35:15.167700 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 06:35:15.167951 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":140,"threads_started":1}
I20260928 06:35:15.170116 6678 heartbeater.cc:347] Connected to a master server at 127.5.22.62:33127
I20260928 06:35:15.170274 6678 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:15.170730 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 6547
I20260928 06:35:15.170838 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:15.170943 6678 heartbeater.cc:511] Master 127.5.22.62:33127 requested a full tablet report, sending...
I20260928 06:35:15.171131 5208 external_mini_cluster.cc:1428] Setting key 2465d8f156405637e21a5f765f8b7b9d
I20260928 06:35:15.171984 6332 ts_manager.cc:194] Registered new tserver with Master: 4656c42c39664c43be519e461ac32ac0 (127.5.22.2:36693)
I20260928 06:35:15.172596 6332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.2@KRBTEST.COM'} at 127.5.22.2:45599
I20260928 06:35:15.180554 6684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:15.172823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:33127 (local address 127.5.22.2:33771)
0928 06:35:15.173099 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:15.173114 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:15.173194 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:15.173479 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:15.173481 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:15.173692 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:15.173787 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:15.173792 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.174617 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.174620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:15.175384 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:15.175397 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.175914 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.175920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:15.176042 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:15.176668 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:15.176678 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:15.176988 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:15.179030 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:15.179033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:15.179045 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:15.179235 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:15.179559 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:15.179562 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:15.179564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:15.179643 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:15.180153 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:15.180155 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:15.180225 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 06:35:15.180304 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:35:15.180404 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":96,"threads_started":1}
I20260928 06:35:15.180706 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:15.173171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:33771 (local address 127.5.22.62:33127)
0928 06:35:15.173286 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:15.173290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:15.173301 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:35:15.173340 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:15.173342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:15.173375 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:15.173463 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:15.173899 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.174489 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.175513 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.175787 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.175910 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:15.177154 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:15.177158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:15.177160 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:15.177191 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:15.178919 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:15.179339 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:15.179345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:15.179349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:15.179449 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:15.179742 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:15.179748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:15.179753 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:15.180040 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:15.180168 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:15.180392 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 06:35:15.180521 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for kudu/127.5.22.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
I20260928 06:35:15.201900 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:33127
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:37869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:15.315994 6688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:15.316300 6688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:15.316373 6688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:15.320487 6688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:15.320580 6688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:15.320649 6688 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:35:15.320735 6688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:35:15.326069 6688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:37869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.22.62:33127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:15.327170 6688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:15.328295 6688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:15.338176 6694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:15.338199 6693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:15.338192 6696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:15.338244 6688 server_base.cc:1034] running on GCE node
I20260928 06:35:15.338913 6688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:15.339625 6688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:15.340826 6688 hybrid_clock.cc:648] HybridClock initialized: now 1790577315340785 us; error 60 us; skew 500 ppm
Sep 28 06:35:15 dist-test-slave-j0qf krb5kdc[6288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577315, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:15.343644 6688 init.cc:377] Logged in from keytab as kudu/127.5.22.3@KRBTEST.COM (short username kudu)
I20260928 06:35:15.344470 6688 webserver.cc:460] Webserver started at http://127.5.22.3:36745/ using document root <none> and password file <none>
I20260928 06:35:15.344985 6688 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:15.345042 6688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:15.345208 6688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:15.347226 6688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "8016ab6336f343e69863237935cc44e5"
format_stamp: "Formatted at 2026-09-28 06:35:15 on dist-test-slave-j0qf"
server_key: "5225126677f0f9ba2938f15531bddb2b"
server_key_iv: "f697b45b6625b94d12d5ff2972b92f01"
server_key_version: "encryptionkey@0"
I20260928 06:35:15.347736 6688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "8016ab6336f343e69863237935cc44e5"
format_stamp: "Formatted at 2026-09-28 06:35:15 on dist-test-slave-j0qf"
server_key: "5225126677f0f9ba2938f15531bddb2b"
server_key_iv: "f697b45b6625b94d12d5ff2972b92f01"
server_key_version: "encryptionkey@0"
I20260928 06:35:15.351357 6688 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:35:15.353670 6703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:15.354979 6688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:35:15.355124 6688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "8016ab6336f343e69863237935cc44e5"
format_stamp: "Formatted at 2026-09-28 06:35:15 on dist-test-slave-j0qf"
server_key: "5225126677f0f9ba2938f15531bddb2b"
server_key_iv: "f697b45b6625b94d12d5ff2972b92f01"
server_key_version: "encryptionkey@0"
I20260928 06:35:15.355235 6688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:15.382899 6688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:15.386210 6688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:15.386445 6688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:15.387853 6688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:15.387928 6688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:15.388000 6688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:15.388046 6688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:35:15 dist-test-slave-j0qf krb5kdc[6288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577315, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:15.399415 6688 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:41125
I20260928 06:35:15.399602 6820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:41125 every 8 connection(s)
I20260928 06:35:15.400540 6688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:15.402779 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:15.388787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:39011 (local address 127.5.22.62:33127)
0928 06:35:15.388923 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:15.388927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:15.389952 (+ 1025us) server_negotiation.cc:407] Connection header received
0928 06:35:15.391122 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:15.391127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:15.391189 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:15.391252 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:15.392652 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.393301 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.394638 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.394864 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.394990 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:15.398566 (+ 3576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:15.398570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:15.398573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:15.398602 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:15.400425 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:15.401104 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:15.401108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:15.401109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:15.401212 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:15.401646 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:15.401648 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:15.401650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:15.401830 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:15.401927 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:15.402487 (+ 560us) server_negotiation.cc:299] Negotiation successful
0928 06:35:15.402593 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:35:15.403676 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:15.389329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:33127 (local address 127.5.22.3:39011)
0928 06:35:15.389801 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:15.389850 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:15.390875 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:15.391717 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:15.391723 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:15.392032 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:15.392506 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:15.392519 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.393430 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.393433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:15.394515 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:15.394524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.395035 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.395039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:15.395195 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:15.395872 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:15.395900 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:15.398396 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:15.400558 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:15.400563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:15.400574 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:15.400961 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:15.401331 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:15.401334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:15.401336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:15.401523 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:15.401945 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:15.401950 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:15.402187 (+ 237us) client_negotiation.cc:769] Sending connection context
0928 06:35:15.402433 (+ 246us) client_negotiation.cc:240] Negotiation successful
0928 06:35:15.402723 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":133,"threads_started":1}
I20260928 06:35:15.405036 6821 heartbeater.cc:347] Connected to a master server at 127.5.22.62:33127
I20260928 06:35:15.405174 6821 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:15.405730 6821 heartbeater.cc:511] Master 127.5.22.62:33127 requested a full tablet report, sending...
I20260928 06:35:15.406852 6333 ts_manager.cc:194] Registered new tserver with Master: 8016ab6336f343e69863237935cc44e5 (127.5.22.3:41125)
I20260928 06:35:15.407505 6333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.3@KRBTEST.COM'} at 127.5.22.3:39011
I20260928 06:35:15.409122 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 6688
I20260928 06:35:15.409250 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:15.409559 5208 external_mini_cluster.cc:1428] Setting key 780f384c5ddad3900312db7f1b97f101
I20260928 06:35:15.411608 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:35:15.415086 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:15.407474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:58029 (local address 127.5.22.62:33127)
0928 06:35:15.407614 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:15.407620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:15.407863 (+ 243us) server_negotiation.cc:407] Connection header received
0928 06:35:15.407979 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:15.407981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:15.408029 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:15.408125 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:15.408666 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.409275 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.410450 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.410704 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.410825 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:15.411959 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:15.411962 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:15.411965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:15.411989 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:15.413405 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:15.413832 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:15.413836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:15.413837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:15.413919 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:15.414200 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:15.414204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:15.414206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:15.414451 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:15.414558 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:15.414818 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 06:35:15.414925 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 06:35:15.415096 6827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:15.407357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:33127 (local address 127.5.22.3:58029)
0928 06:35:15.407773 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:15.407788 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:15.407883 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:15.408164 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:15.408167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:15.408372 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:15.408523 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:15.408531 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.409429 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.409432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:15.410307 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:15.410317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.410829 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.410833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:15.410924 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:15.411440 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:15.411452 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:15.411833 (+ 381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:15.413520 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:15.413524 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:15.413526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:15.413726 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:15.414019 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:15.414022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:15.414025 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:15.414110 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:15.414556 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:15.414559 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:15.414655 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 06:35:15.414765 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 06:35:15.414901 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":1,"thread_start_us":141,"threads_started":1}
Sep 28 06:35:15 dist-test-slave-j0qf krb5kdc[6288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577315, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:35:15 dist-test-slave-j0qf krb5kdc[6288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577315, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:15.453675 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:15.438536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46250 (local address 127.5.22.62:33127)
0928 06:35:15.438729 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:15.438735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:15.438755 (+ 20us) server_negotiation.cc:407] Connection header received
0928 06:35:15.438817 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:15.438820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:15.438881 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:15.438993 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:15.439883 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.440849 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.442292 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.442833 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.442939 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:15.450241 (+ 7302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:15.450246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:15.450248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:15.450286 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:15.451905 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:15.452362 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:15.452366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:15.452368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:15.452466 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:15.452748 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:15.452752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:15.452754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:15.452948 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:15.453088 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:15.453358 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 06:35:15.453497 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 06:35:15.457407 6333 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:35:15.460839 6333 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:15.477051 6838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:15.470228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:41125 (local address 127.0.0.1:45354)
0928 06:35:15.470604 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:15.470652 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:15.470776 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:15.472240 (+ 1464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:15.472247 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:15.472270 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:15.472452 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:15.472459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.473820 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.473823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:15.475199 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:15.475209 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.476487 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.476490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:15.476592 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:15.476639 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 06:35:15.476755 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 06:35:15.476889 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":134,"threads_started":1}
I20260928 06:35:15.477349 6842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:15.472021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:39565 (local address 127.0.0.1:45662)
0928 06:35:15.472274 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:15.472287 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:15.472363 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:15.473206 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:15.473209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:15.473226 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:15.473306 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:15.473311 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.474773 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.474776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:15.475878 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:15.475888 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.476954 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.476957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:15.477037 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:15.477059 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 06:35:15.477111 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:35:15.477200 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":3,"thread_start_us":92,"threads_started":1}
I20260928 06:35:15.477555 6840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:15.471072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36794 (local address 127.5.22.2:36693)
0928 06:35:15.471536 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:15.471549 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:15.472237 (+ 688us) server_negotiation.cc:407] Connection header received
0928 06:35:15.472315 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:15.472362 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:15.472620 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:15.472764 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:15.473105 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.474164 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.475725 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.476557 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.476667 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:15.476810 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:15.477275 (+ 465us) server_negotiation.cc:299] Negotiation successful
0928 06:35:15.477371 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":130,"threads_started":1}
I20260928 06:35:15.477947 6841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:15.471083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45354 (local address 127.5.22.3:41125)
0928 06:35:15.471535 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:15.471549 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:15.471576 (+ 27us) server_negotiation.cc:407] Connection header received
0928 06:35:15.471658 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:15.471663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:15.471875 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:15.472029 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:15.472576 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.473694 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.475335 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.476161 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.476256 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:15.476413 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:15.477626 (+ 1213us) server_negotiation.cc:299] Negotiation successful
0928 06:35:15.477810 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":116,"threads_started":1}
I20260928 06:35:15.477974 6843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:15.472444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45662 (local address 127.5.22.1:39565)
0928 06:35:15.472784 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:15.472791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:15.472812 (+ 21us) server_negotiation.cc:407] Connection header received
0928 06:35:15.472881 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:15.472886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:15.473063 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:15.473437 (+ 374us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:15.473550 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.474647 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.476042 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.476817 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.477588 (+ 771us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:15.477722 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:15.477765 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 06:35:15.477841 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":99,"threads_started":1}
I20260928 06:35:15.478879 6839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:15.470275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:36693 (local address 127.0.0.1:36794)
0928 06:35:15.472028 (+ 1753us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:15.472041 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:15.472139 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:15.472889 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:15.472891 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:15.472905 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:15.472989 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:15.472993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.474294 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.474297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:15.475568 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:15.475578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.476872 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.476877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:15.476979 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:15.477010 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 06:35:15.477080 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 06:35:15.477160 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1682,"mutex_wait_us":1,"thread_start_us":156,"threads_started":1}
I20260928 06:35:15.481148 6612 tablet_service.cc:1467] Processing CreateTablet for tablet b2a848ab79294d3c876a1cb38974013c (DEFAULT_TABLE table=test-table [id=a33363f5f35a49c5ac5e2dd6a13ac457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:15.481135 6471 tablet_service.cc:1467] Processing CreateTablet for tablet b2a848ab79294d3c876a1cb38974013c (DEFAULT_TABLE table=test-table [id=a33363f5f35a49c5ac5e2dd6a13ac457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:15.482226 6612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2a848ab79294d3c876a1cb38974013c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:15.482261 6471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2a848ab79294d3c876a1cb38974013c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:15.484344 6754 tablet_service.cc:1467] Processing CreateTablet for tablet b2a848ab79294d3c876a1cb38974013c (DEFAULT_TABLE table=test-table [id=a33363f5f35a49c5ac5e2dd6a13ac457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:15.485358 6754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2a848ab79294d3c876a1cb38974013c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:15.490061 6844 tablet_bootstrap.cc:492] T b2a848ab79294d3c876a1cb38974013c P 5241ea335d87483b85f2a0e0c9d727ca: Bootstrap starting.
I20260928 06:35:15.492149 6844 tablet_bootstrap.cc:654] T b2a848ab79294d3c876a1cb38974013c P 5241ea335d87483b85f2a0e0c9d727ca: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:15.492733 6846 tablet_bootstrap.cc:492] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5: Bootstrap starting.
I20260928 06:35:15.492928 6844 log.cc:824] T b2a848ab79294d3c876a1cb38974013c P 5241ea335d87483b85f2a0e0c9d727ca: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:15.493959 6845 tablet_bootstrap.cc:492] T b2a848ab79294d3c876a1cb38974013c P 4656c42c39664c43be519e461ac32ac0: Bootstrap starting.
I20260928 06:35:15.494984 6844 tablet_bootstrap.cc:492] T b2a848ab79294d3c876a1cb38974013c P 5241ea335d87483b85f2a0e0c9d727ca: No bootstrap required, opened a new log
I20260928 06:35:15.495025 6846 tablet_bootstrap.cc:654] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:15.495159 6844 ts_tablet_manager.cc:1397] T b2a848ab79294d3c876a1cb38974013c P 5241ea335d87483b85f2a0e0c9d727ca: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 06:35:15.495885 6846 log.cc:824] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:15.496001 6845 tablet_bootstrap.cc:654] T b2a848ab79294d3c876a1cb38974013c P 4656c42c39664c43be519e461ac32ac0: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:15.496747 6845 log.cc:824] T b2a848ab79294d3c876a1cb38974013c P 4656c42c39664c43be519e461ac32ac0: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:15.499074 6845 tablet_bootstrap.cc:492] T b2a848ab79294d3c876a1cb38974013c P 4656c42c39664c43be519e461ac32ac0: No bootstrap required, opened a new log
I20260928 06:35:15.499318 6845 ts_tablet_manager.cc:1397] T b2a848ab79294d3c876a1cb38974013c P 4656c42c39664c43be519e461ac32ac0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 06:35:15.499577 6844 raft_consensus.cc:348] T b2a848ab79294d3c876a1cb38974013c P 5241ea335d87483b85f2a0e0c9d727ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } }
I20260928 06:35:15.499867 6844 raft_consensus.cc:374] T b2a848ab79294d3c876a1cb38974013c P 5241ea335d87483b85f2a0e0c9d727ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:15.499920 6844 raft_consensus.cc:729] T b2a848ab79294d3c876a1cb38974013c P 5241ea335d87483b85f2a0e0c9d727ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5241ea335d87483b85f2a0e0c9d727ca, State: Initialized, Role: FOLLOWER
I20260928 06:35:15.500030 6846 tablet_bootstrap.cc:492] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5: No bootstrap required, opened a new log
I20260928 06:35:15.500226 6846 ts_tablet_manager.cc:1397] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260928 06:35:15.500429 6844 consensus_queue.cc:260] T b2a848ab79294d3c876a1cb38974013c P 5241ea335d87483b85f2a0e0c9d727ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } }
I20260928 06:35:15.501714 6844 ts_tablet_manager.cc:1428] T b2a848ab79294d3c876a1cb38974013c P 5241ea335d87483b85f2a0e0c9d727ca: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 06:35:15.502508 6538 heartbeater.cc:503] Master 127.5.22.62:33127 was elected leader, sending a full tablet report...
I20260928 06:35:15.504421 6846 raft_consensus.cc:348] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } }
I20260928 06:35:15.504714 6846 raft_consensus.cc:374] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:15.504773 6846 raft_consensus.cc:729] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8016ab6336f343e69863237935cc44e5, State: Initialized, Role: FOLLOWER
I20260928 06:35:15.505330 6846 consensus_queue.cc:260] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } }
I20260928 06:35:15.504213 6845 raft_consensus.cc:348] T b2a848ab79294d3c876a1cb38974013c P 4656c42c39664c43be519e461ac32ac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } }
I20260928 06:35:15.505566 6845 raft_consensus.cc:374] T b2a848ab79294d3c876a1cb38974013c P 4656c42c39664c43be519e461ac32ac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:15.505637 6845 raft_consensus.cc:729] T b2a848ab79294d3c876a1cb38974013c P 4656c42c39664c43be519e461ac32ac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4656c42c39664c43be519e461ac32ac0, State: Initialized, Role: FOLLOWER
I20260928 06:35:15.506419 6845 consensus_queue.cc:260] T b2a848ab79294d3c876a1cb38974013c P 4656c42c39664c43be519e461ac32ac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } }
I20260928 06:35:15.506568 6846 ts_tablet_manager.cc:1428] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 06:35:15.506945 6821 heartbeater.cc:503] Master 127.5.22.62:33127 was elected leader, sending a full tablet report...
I20260928 06:35:15.507082 6678 heartbeater.cc:503] Master 127.5.22.62:33127 was elected leader, sending a full tablet report...
I20260928 06:35:15.509367 6845 ts_tablet_manager.cc:1428] T b2a848ab79294d3c876a1cb38974013c P 4656c42c39664c43be519e461ac32ac0: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
W20260928 06:35:15.651605 6822 tablet.cc:2369] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:35:15.667186 6679 tablet.cc:2369] T b2a848ab79294d3c876a1cb38974013c P 4656c42c39664c43be519e461ac32ac0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:35:15.687004 6539 tablet.cc:2369] T b2a848ab79294d3c876a1cb38974013c P 5241ea335d87483b85f2a0e0c9d727ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:35:15.717001 6851 raft_consensus.cc:482] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:15.717247 6851 raft_consensus.cc:504] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } }
I20260928 06:35:15.718500 6851 leader_election.cc:288] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4656c42c39664c43be519e461ac32ac0 (127.5.22.2:36693), 5241ea335d87483b85f2a0e0c9d727ca (127.5.22.1:39565)
I20260928 06:35:15.722188 6840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:15.718854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:48009 (local address 127.5.22.2:36693)
0928 06:35:15.719008 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:15.719014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:15.719032 (+ 18us) server_negotiation.cc:407] Connection header received
0928 06:35:15.719198 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:15.719204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:15.719262 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:15.719367 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:15.719638 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.720144 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.721178 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.721567 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.721682 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:15.721746 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:15.721993 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 06:35:15.722077 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 06:35:15.722188 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:15.718761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:36693 (local address 127.5.22.3:48009)
0928 06:35:15.718909 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:15.718928 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:15.719066 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:15.719380 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:15.719383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:15.719400 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:15.719514 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:15.719525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.720255 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.720261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:15.721051 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:15.721059 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.721691 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.721695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:15.721870 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:15.721887 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 06:35:15.721976 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:35:15.722051 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":48}
I20260928 06:35:15.722919 6632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a848ab79294d3c876a1cb38974013c" candidate_uuid: "8016ab6336f343e69863237935cc44e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4656c42c39664c43be519e461ac32ac0" is_pre_election: true
I20260928 06:35:15.723143 6853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:15.719029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:39565 (local address 127.5.22.3:45277)
0928 06:35:15.719393 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:15.719406 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:15.719526 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:15.719772 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:15.719774 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:15.719789 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:15.720276 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:15.720283 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.721015 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.721021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:15.721835 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:15.721844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.722698 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.722702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:15.722806 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:15.722827 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 06:35:15.722906 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:35:15.722973 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":22,"thread_start_us":153,"threads_started":1}
I20260928 06:35:15.723148 6843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:15.719208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:45277 (local address 127.5.22.1:39565)
0928 06:35:15.719367 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:15.719372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:15.719566 (+ 194us) server_negotiation.cc:407] Connection header received
0928 06:35:15.719626 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:15.719628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:15.719677 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:15.719761 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:15.720407 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.720900 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.721970 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.722558 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.722675 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:15.722720 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:15.722932 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 06:35:15.722998 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 06:35:15.723254 6632 raft_consensus.cc:2454] T b2a848ab79294d3c876a1cb38974013c P 4656c42c39664c43be519e461ac32ac0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8016ab6336f343e69863237935cc44e5 in term 0.
I20260928 06:35:15.723779 6491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a848ab79294d3c876a1cb38974013c" candidate_uuid: "8016ab6336f343e69863237935cc44e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5241ea335d87483b85f2a0e0c9d727ca" is_pre_election: true
I20260928 06:35:15.723825 6706 leader_election.cc:302] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4656c42c39664c43be519e461ac32ac0, 8016ab6336f343e69863237935cc44e5; no voters:
I20260928 06:35:15.724092 6851 raft_consensus.cc:2781] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:15.724093 6491 raft_consensus.cc:2454] T b2a848ab79294d3c876a1cb38974013c P 5241ea335d87483b85f2a0e0c9d727ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8016ab6336f343e69863237935cc44e5 in term 0.
I20260928 06:35:15.724176 6851 raft_consensus.cc:482] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:15.724211 6851 raft_consensus.cc:3037] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:15.725370 6851 raft_consensus.cc:504] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } }
I20260928 06:35:15.725715 6851 leader_election.cc:288] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5 [CANDIDATE]: Term 1 election: Requested vote from peers 4656c42c39664c43be519e461ac32ac0 (127.5.22.2:36693), 5241ea335d87483b85f2a0e0c9d727ca (127.5.22.1:39565)
I20260928 06:35:15.726240 6632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a848ab79294d3c876a1cb38974013c" candidate_uuid: "8016ab6336f343e69863237935cc44e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4656c42c39664c43be519e461ac32ac0"
I20260928 06:35:15.726286 6491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a848ab79294d3c876a1cb38974013c" candidate_uuid: "8016ab6336f343e69863237935cc44e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5241ea335d87483b85f2a0e0c9d727ca"
I20260928 06:35:15.726394 6632 raft_consensus.cc:3037] T b2a848ab79294d3c876a1cb38974013c P 4656c42c39664c43be519e461ac32ac0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:15.726411 6491 raft_consensus.cc:3037] T b2a848ab79294d3c876a1cb38974013c P 5241ea335d87483b85f2a0e0c9d727ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:15.727730 6491 raft_consensus.cc:2454] T b2a848ab79294d3c876a1cb38974013c P 5241ea335d87483b85f2a0e0c9d727ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8016ab6336f343e69863237935cc44e5 in term 1.
I20260928 06:35:15.727730 6632 raft_consensus.cc:2454] T b2a848ab79294d3c876a1cb38974013c P 4656c42c39664c43be519e461ac32ac0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8016ab6336f343e69863237935cc44e5 in term 1.
I20260928 06:35:15.728013 6706 leader_election.cc:302] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4656c42c39664c43be519e461ac32ac0, 8016ab6336f343e69863237935cc44e5; no voters:
I20260928 06:35:15.728214 6851 raft_consensus.cc:2781] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:15.728437 6851 raft_consensus.cc:686] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5 [term 1 LEADER]: Becoming Leader. State: Replica: 8016ab6336f343e69863237935cc44e5, State: Running, Role: LEADER
I20260928 06:35:15.728770 6851 consensus_queue.cc:237] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } }
I20260928 06:35:15.731976 6333 catalog_manager.cc:5495] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8016ab6336f343e69863237935cc44e5 (127.5.22.3). New cstate: current_term: 1 leader_uuid: "8016ab6336f343e69863237935cc44e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } health_report { overall_health: UNKNOWN } } }
I20260928 06:35:15.780566 6841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:15.776248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45370 (local address 127.5.22.3:41125)
0928 06:35:15.776399 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:15.776404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:15.776420 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:35:15.776509 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:15.776514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:15.776589 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:15.776702 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:15.777143 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.777990 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.779051 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.779257 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.779336 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:15.779701 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:15.780086 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:15.780171 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:15.780321 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 06:35:15.780394 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 06:35:15.786682 6491 raft_consensus.cc:1261] T b2a848ab79294d3c876a1cb38974013c P 5241ea335d87483b85f2a0e0c9d727ca [term 1 FOLLOWER]: Refusing update from remote peer 8016ab6336f343e69863237935cc44e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:15.786703 6632 raft_consensus.cc:1261] T b2a848ab79294d3c876a1cb38974013c P 4656c42c39664c43be519e461ac32ac0 [term 1 FOLLOWER]: Refusing update from remote peer 8016ab6336f343e69863237935cc44e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:15.787449 6851 consensus_queue.cc:1035] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:15.787690 6854 consensus_queue.cc:1035] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:15.809746 6332 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:15.809931 6332 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:15.812659 6332 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 179eaae02aec40a48344f40bf270f9d8: Sending DeleteTablet for 3 replicas of tablet b2a848ab79294d3c876a1cb38974013c
I20260928 06:35:15.813494 6754 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a848ab79294d3c876a1cb38974013c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:15 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:45354
I20260928 06:35:15.813606 6612 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a848ab79294d3c876a1cb38974013c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:15 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:36794
I20260928 06:35:15.813776 6471 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a848ab79294d3c876a1cb38974013c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:15 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:45662
I20260928 06:35:15.814083 6867 tablet_replica.cc:330] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5: stopping tablet replica
I20260928 06:35:15.814361 6868 tablet_replica.cc:330] T b2a848ab79294d3c876a1cb38974013c P 4656c42c39664c43be519e461ac32ac0: stopping tablet replica
I20260928 06:35:15.814514 6867 raft_consensus.cc:2229] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:35:15.814693 6868 raft_consensus.cc:2229] T b2a848ab79294d3c876a1cb38974013c P 4656c42c39664c43be519e461ac32ac0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:15.814888 6332 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 06:35:15.815027 6868 raft_consensus.cc:2258] T b2a848ab79294d3c876a1cb38974013c P 4656c42c39664c43be519e461ac32ac0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:15.815222 6867 raft_consensus.cc:2258] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 06:35:15.815351 6332 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:15.816318 6869 tablet_replica.cc:330] T b2a848ab79294d3c876a1cb38974013c P 5241ea335d87483b85f2a0e0c9d727ca: stopping tablet replica
I20260928 06:35:15.816627 6869 raft_consensus.cc:2229] T b2a848ab79294d3c876a1cb38974013c P 5241ea335d87483b85f2a0e0c9d727ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:15.816946 6869 raft_consensus.cc:2258] T b2a848ab79294d3c876a1cb38974013c P 5241ea335d87483b85f2a0e0c9d727ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:15.817533 6868 ts_tablet_manager.cc:1905] T b2a848ab79294d3c876a1cb38974013c P 4656c42c39664c43be519e461ac32ac0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:15.818835 6867 ts_tablet_manager.cc:1905] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:15.820573 6868 ts_tablet_manager.cc:1918] T b2a848ab79294d3c876a1cb38974013c P 4656c42c39664c43be519e461ac32ac0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:15.820762 6868 log.cc:1196] T b2a848ab79294d3c876a1cb38974013c P 4656c42c39664c43be519e461ac32ac0: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-1/wal/wals/b2a848ab79294d3c876a1cb38974013c
I20260928 06:35:15.821192 6868 ts_tablet_manager.cc:1939] T b2a848ab79294d3c876a1cb38974013c P 4656c42c39664c43be519e461ac32ac0: Deleting consensus metadata
I20260928 06:35:15.821322 6867 ts_tablet_manager.cc:1918] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:15.821415 6867 log.cc:1196] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/b2a848ab79294d3c876a1cb38974013c
I20260928 06:35:15.821710 6867 ts_tablet_manager.cc:1939] T b2a848ab79294d3c876a1cb38974013c P 8016ab6336f343e69863237935cc44e5: Deleting consensus metadata
I20260928 06:35:15.822827 6319 catalog_manager.cc:4841] TS 4656c42c39664c43be519e461ac32ac0 (127.5.22.2:36693): tablet b2a848ab79294d3c876a1cb38974013c (table test-table [id=a33363f5f35a49c5ac5e2dd6a13ac457]) successfully deleted
I20260928 06:35:15.822993 6754 tablet_service.cc:1467] Processing CreateTablet for tablet fb90c978a1524e7ba5e4ccee37dd790f (DEFAULT_TABLE table=test-table [id=d57cf6274c0e496d80d79e39b398867c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:15.823103 6471 tablet_service.cc:1467] Processing CreateTablet for tablet fb90c978a1524e7ba5e4ccee37dd790f (DEFAULT_TABLE table=test-table [id=d57cf6274c0e496d80d79e39b398867c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:15.823235 6754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb90c978a1524e7ba5e4ccee37dd790f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:15.823413 6471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb90c978a1524e7ba5e4ccee37dd790f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:15.825241 6844 tablet_bootstrap.cc:492] T fb90c978a1524e7ba5e4ccee37dd790f P 5241ea335d87483b85f2a0e0c9d727ca: Bootstrap starting.
I20260928 06:35:15.825459 6612 tablet_service.cc:1467] Processing CreateTablet for tablet fb90c978a1524e7ba5e4ccee37dd790f (DEFAULT_TABLE table=test-table [id=d57cf6274c0e496d80d79e39b398867c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:15.825687 6612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb90c978a1524e7ba5e4ccee37dd790f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:15.826260 6844 tablet_bootstrap.cc:654] T fb90c978a1524e7ba5e4ccee37dd790f P 5241ea335d87483b85f2a0e0c9d727ca: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:15.827593 6844 tablet_bootstrap.cc:492] T fb90c978a1524e7ba5e4ccee37dd790f P 5241ea335d87483b85f2a0e0c9d727ca: No bootstrap required, opened a new log
I20260928 06:35:15.827690 6844 ts_tablet_manager.cc:1397] T fb90c978a1524e7ba5e4ccee37dd790f P 5241ea335d87483b85f2a0e0c9d727ca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:15.827987 6845 tablet_bootstrap.cc:492] T fb90c978a1524e7ba5e4ccee37dd790f P 4656c42c39664c43be519e461ac32ac0: Bootstrap starting.
I20260928 06:35:15.828063 6846 tablet_bootstrap.cc:492] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5: Bootstrap starting.
I20260928 06:35:15.828172 6844 raft_consensus.cc:348] T fb90c978a1524e7ba5e4ccee37dd790f P 5241ea335d87483b85f2a0e0c9d727ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } }
I20260928 06:35:15.828270 6844 raft_consensus.cc:374] T fb90c978a1524e7ba5e4ccee37dd790f P 5241ea335d87483b85f2a0e0c9d727ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:15.828303 6844 raft_consensus.cc:729] T fb90c978a1524e7ba5e4ccee37dd790f P 5241ea335d87483b85f2a0e0c9d727ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5241ea335d87483b85f2a0e0c9d727ca, State: Initialized, Role: FOLLOWER
I20260928 06:35:15.828438 6844 consensus_queue.cc:260] T fb90c978a1524e7ba5e4ccee37dd790f P 5241ea335d87483b85f2a0e0c9d727ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } }
I20260928 06:35:15.828902 6845 tablet_bootstrap.cc:654] T fb90c978a1524e7ba5e4ccee37dd790f P 4656c42c39664c43be519e461ac32ac0: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:15.829021 6846 tablet_bootstrap.cc:654] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:15.829579 6320 catalog_manager.cc:4841] TS 8016ab6336f343e69863237935cc44e5 (127.5.22.3:41125): tablet b2a848ab79294d3c876a1cb38974013c (table test-table [id=a33363f5f35a49c5ac5e2dd6a13ac457]) successfully deleted
I20260928 06:35:15.829716 6844 ts_tablet_manager.cc:1428] T fb90c978a1524e7ba5e4ccee37dd790f P 5241ea335d87483b85f2a0e0c9d727ca: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:35:15.831439 6846 tablet_bootstrap.cc:492] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5: No bootstrap required, opened a new log
I20260928 06:35:15.831552 6846 ts_tablet_manager.cc:1397] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 06:35:15.832019 6846 raft_consensus.cc:348] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } }
I20260928 06:35:15.832156 6846 raft_consensus.cc:374] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:15.832191 6846 raft_consensus.cc:729] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8016ab6336f343e69863237935cc44e5, State: Initialized, Role: FOLLOWER
I20260928 06:35:15.832325 6846 consensus_queue.cc:260] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } }
I20260928 06:35:15.832682 6846 ts_tablet_manager.cc:1428] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:15.834295 6869 ts_tablet_manager.cc:1905] T b2a848ab79294d3c876a1cb38974013c P 5241ea335d87483b85f2a0e0c9d727ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:15.837446 6869 ts_tablet_manager.cc:1918] T b2a848ab79294d3c876a1cb38974013c P 5241ea335d87483b85f2a0e0c9d727ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:15.837543 6869 log.cc:1196] T b2a848ab79294d3c876a1cb38974013c P 5241ea335d87483b85f2a0e0c9d727ca: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-0/wal/wals/b2a848ab79294d3c876a1cb38974013c
I20260928 06:35:15.837845 6869 ts_tablet_manager.cc:1939] T b2a848ab79294d3c876a1cb38974013c P 5241ea335d87483b85f2a0e0c9d727ca: Deleting consensus metadata
I20260928 06:35:15.839071 6319 catalog_manager.cc:4841] TS 5241ea335d87483b85f2a0e0c9d727ca (127.5.22.1:39565): tablet b2a848ab79294d3c876a1cb38974013c (table test-table [id=a33363f5f35a49c5ac5e2dd6a13ac457]) successfully deleted
I20260928 06:35:15.839787 6845 tablet_bootstrap.cc:492] T fb90c978a1524e7ba5e4ccee37dd790f P 4656c42c39664c43be519e461ac32ac0: No bootstrap required, opened a new log
I20260928 06:35:15.839890 6845 ts_tablet_manager.cc:1397] T fb90c978a1524e7ba5e4ccee37dd790f P 4656c42c39664c43be519e461ac32ac0: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260928 06:35:15.840400 6845 raft_consensus.cc:348] T fb90c978a1524e7ba5e4ccee37dd790f P 4656c42c39664c43be519e461ac32ac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } }
I20260928 06:35:15.840524 6845 raft_consensus.cc:374] T fb90c978a1524e7ba5e4ccee37dd790f P 4656c42c39664c43be519e461ac32ac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:15.840557 6845 raft_consensus.cc:729] T fb90c978a1524e7ba5e4ccee37dd790f P 4656c42c39664c43be519e461ac32ac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4656c42c39664c43be519e461ac32ac0, State: Initialized, Role: FOLLOWER
I20260928 06:35:15.840677 6845 consensus_queue.cc:260] T fb90c978a1524e7ba5e4ccee37dd790f P 4656c42c39664c43be519e461ac32ac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } }
I20260928 06:35:15.840922 6845 ts_tablet_manager.cc:1428] T fb90c978a1524e7ba5e4ccee37dd790f P 4656c42c39664c43be519e461ac32ac0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:35:15.854985 6854 raft_consensus.cc:482] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:15.855127 6854 raft_consensus.cc:504] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } }
I20260928 06:35:15.855567 6854 leader_election.cc:288] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5241ea335d87483b85f2a0e0c9d727ca (127.5.22.1:39565), 4656c42c39664c43be519e461ac32ac0 (127.5.22.2:36693)
I20260928 06:35:15.856027 6632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb90c978a1524e7ba5e4ccee37dd790f" candidate_uuid: "8016ab6336f343e69863237935cc44e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4656c42c39664c43be519e461ac32ac0" is_pre_election: true
I20260928 06:35:15.855983 6491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb90c978a1524e7ba5e4ccee37dd790f" candidate_uuid: "8016ab6336f343e69863237935cc44e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5241ea335d87483b85f2a0e0c9d727ca" is_pre_election: true
I20260928 06:35:15.856173 6632 raft_consensus.cc:2454] T fb90c978a1524e7ba5e4ccee37dd790f P 4656c42c39664c43be519e461ac32ac0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8016ab6336f343e69863237935cc44e5 in term 0.
I20260928 06:35:15.856225 6491 raft_consensus.cc:2454] T fb90c978a1524e7ba5e4ccee37dd790f P 5241ea335d87483b85f2a0e0c9d727ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8016ab6336f343e69863237935cc44e5 in term 0.
I20260928 06:35:15.856465 6706 leader_election.cc:302] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4656c42c39664c43be519e461ac32ac0, 8016ab6336f343e69863237935cc44e5; no voters:
I20260928 06:35:15.856660 6854 raft_consensus.cc:2781] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:15.856729 6854 raft_consensus.cc:482] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:15.856767 6854 raft_consensus.cc:3037] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:15.858022 6854 raft_consensus.cc:504] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } }
I20260928 06:35:15.858400 6854 leader_election.cc:288] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5 [CANDIDATE]: Term 1 election: Requested vote from peers 5241ea335d87483b85f2a0e0c9d727ca (127.5.22.1:39565), 4656c42c39664c43be519e461ac32ac0 (127.5.22.2:36693)
I20260928 06:35:15.858865 6491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb90c978a1524e7ba5e4ccee37dd790f" candidate_uuid: "8016ab6336f343e69863237935cc44e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5241ea335d87483b85f2a0e0c9d727ca"
I20260928 06:35:15.858877 6632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb90c978a1524e7ba5e4ccee37dd790f" candidate_uuid: "8016ab6336f343e69863237935cc44e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4656c42c39664c43be519e461ac32ac0"
I20260928 06:35:15.858984 6491 raft_consensus.cc:3037] T fb90c978a1524e7ba5e4ccee37dd790f P 5241ea335d87483b85f2a0e0c9d727ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:15.859019 6632 raft_consensus.cc:3037] T fb90c978a1524e7ba5e4ccee37dd790f P 4656c42c39664c43be519e461ac32ac0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:15.860136 6632 raft_consensus.cc:2454] T fb90c978a1524e7ba5e4ccee37dd790f P 4656c42c39664c43be519e461ac32ac0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8016ab6336f343e69863237935cc44e5 in term 1.
I20260928 06:35:15.860137 6491 raft_consensus.cc:2454] T fb90c978a1524e7ba5e4ccee37dd790f P 5241ea335d87483b85f2a0e0c9d727ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8016ab6336f343e69863237935cc44e5 in term 1.
I20260928 06:35:15.860443 6706 leader_election.cc:302] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5241ea335d87483b85f2a0e0c9d727ca, 8016ab6336f343e69863237935cc44e5; no voters:
I20260928 06:35:15.860656 6854 raft_consensus.cc:2781] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:15.860754 6854 raft_consensus.cc:686] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5 [term 1 LEADER]: Becoming Leader. State: Replica: 8016ab6336f343e69863237935cc44e5, State: Running, Role: LEADER
I20260928 06:35:15.860883 6854 consensus_queue.cc:237] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } }
I20260928 06:35:15.862610 6332 catalog_manager.cc:5495] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8016ab6336f343e69863237935cc44e5 (127.5.22.3). New cstate: current_term: 1 leader_uuid: "8016ab6336f343e69863237935cc44e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } health_report { overall_health: HEALTHY } } }
Sep 28 06:35:15 dist-test-slave-j0qf krb5kdc[6288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577314, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:15.897504 6879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:15.888174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:33127 (local address 127.0.0.1:46262)
0928 06:35:15.888497 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:15.888513 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:15.888671 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:15.888979 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:15.888981 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:15.889396 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:15.889498 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:15.889504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.890600 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.890602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:15.891171 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:15.891179 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.891750 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.891753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:15.891819 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:15.892456 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:15.892469 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:15.894098 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:15.895992 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:15.896001 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:15.896006 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:15.896245 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:15.896531 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:15.896533 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:15.896541 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:15.896636 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:15.897049 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:15.897056 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:15.897163 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 06:35:15.897255 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 06:35:15.897368 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":104,"threads_started":1}
I20260928 06:35:15.897639 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:15.888291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46262 (local address 127.5.22.62:33127)
0928 06:35:15.888455 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:15.888460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:15.888611 (+ 151us) server_negotiation.cc:407] Connection header received
0928 06:35:15.888790 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:15.888793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:15.888846 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:15.888944 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:15.889627 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.890477 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.891293 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.891631 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.891736 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:15.894236 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:15.894240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:15.894243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:15.894269 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:15.895873 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:15.896340 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:15.896344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:15.896345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:15.896434 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:15.896724 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:15.896728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:15.896729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:15.896946 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:15.897078 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:15.897333 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 06:35:15.897451 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 06:35:15.901726 6332 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:46262:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 06:35:15.902593 6332 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:15.907720 6471 tablet_service.cc:1467] Processing CreateTablet for tablet 6d9c9218aa7f454d91786db1d6b07fdc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=46d8f80cfdf84d79a3a7211b33855b79]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:35:15.907771 6754 tablet_service.cc:1467] Processing CreateTablet for tablet 6d9c9218aa7f454d91786db1d6b07fdc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=46d8f80cfdf84d79a3a7211b33855b79]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:35:15.907795 6612 tablet_service.cc:1467] Processing CreateTablet for tablet 6d9c9218aa7f454d91786db1d6b07fdc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=46d8f80cfdf84d79a3a7211b33855b79]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:35:15.908139 6612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d9c9218aa7f454d91786db1d6b07fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:15.908139 6471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d9c9218aa7f454d91786db1d6b07fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:15.908162 6754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d9c9218aa7f454d91786db1d6b07fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:15.910933 6846 tablet_bootstrap.cc:492] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5: Bootstrap starting.
I20260928 06:35:15.911387 6844 tablet_bootstrap.cc:492] T 6d9c9218aa7f454d91786db1d6b07fdc P 5241ea335d87483b85f2a0e0c9d727ca: Bootstrap starting.
I20260928 06:35:15.911832 6846 tablet_bootstrap.cc:654] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:15.912346 6844 tablet_bootstrap.cc:654] T 6d9c9218aa7f454d91786db1d6b07fdc P 5241ea335d87483b85f2a0e0c9d727ca: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:15.913205 6846 tablet_bootstrap.cc:492] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5: No bootstrap required, opened a new log
I20260928 06:35:15.913295 6846 ts_tablet_manager.cc:1397] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:15.913673 6846 raft_consensus.cc:348] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } }
I20260928 06:35:15.913805 6846 raft_consensus.cc:374] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:15.913847 6846 raft_consensus.cc:729] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8016ab6336f343e69863237935cc44e5, State: Initialized, Role: FOLLOWER
I20260928 06:35:15.913988 6846 consensus_queue.cc:260] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } }
I20260928 06:35:15.914289 6846 ts_tablet_manager.cc:1428] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:35:15.914285 6854 tablet_replica.cc:439] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } } }
I20260928 06:35:15.914521 6854 tablet_replica.cc:442] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:15.914738 6845 tablet_bootstrap.cc:492] T 6d9c9218aa7f454d91786db1d6b07fdc P 4656c42c39664c43be519e461ac32ac0: Bootstrap starting.
I20260928 06:35:15.915323 6844 tablet_bootstrap.cc:492] T 6d9c9218aa7f454d91786db1d6b07fdc P 5241ea335d87483b85f2a0e0c9d727ca: No bootstrap required, opened a new log
I20260928 06:35:15.915431 6844 ts_tablet_manager.cc:1397] T 6d9c9218aa7f454d91786db1d6b07fdc P 5241ea335d87483b85f2a0e0c9d727ca: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 06:35:15.915710 6845 tablet_bootstrap.cc:654] T 6d9c9218aa7f454d91786db1d6b07fdc P 4656c42c39664c43be519e461ac32ac0: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:15.915822 6844 raft_consensus.cc:348] T 6d9c9218aa7f454d91786db1d6b07fdc P 5241ea335d87483b85f2a0e0c9d727ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } }
I20260928 06:35:15.915941 6844 raft_consensus.cc:374] T 6d9c9218aa7f454d91786db1d6b07fdc P 5241ea335d87483b85f2a0e0c9d727ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:15.915980 6844 raft_consensus.cc:729] T 6d9c9218aa7f454d91786db1d6b07fdc P 5241ea335d87483b85f2a0e0c9d727ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5241ea335d87483b85f2a0e0c9d727ca, State: Initialized, Role: FOLLOWER
I20260928 06:35:15.916119 6844 consensus_queue.cc:260] T 6d9c9218aa7f454d91786db1d6b07fdc P 5241ea335d87483b85f2a0e0c9d727ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } }
I20260928 06:35:15.916400 6844 ts_tablet_manager.cc:1428] T 6d9c9218aa7f454d91786db1d6b07fdc P 5241ea335d87483b85f2a0e0c9d727ca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:15.916669 6850 tablet_replica.cc:439] T 6d9c9218aa7f454d91786db1d6b07fdc P 5241ea335d87483b85f2a0e0c9d727ca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } } }
I20260928 06:35:15.916796 6850 tablet_replica.cc:442] T 6d9c9218aa7f454d91786db1d6b07fdc P 5241ea335d87483b85f2a0e0c9d727ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:15.918720 6845 tablet_bootstrap.cc:492] T 6d9c9218aa7f454d91786db1d6b07fdc P 4656c42c39664c43be519e461ac32ac0: No bootstrap required, opened a new log
I20260928 06:35:15.918805 6845 ts_tablet_manager.cc:1397] T 6d9c9218aa7f454d91786db1d6b07fdc P 4656c42c39664c43be519e461ac32ac0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 06:35:15.919227 6845 raft_consensus.cc:348] T 6d9c9218aa7f454d91786db1d6b07fdc P 4656c42c39664c43be519e461ac32ac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } }
I20260928 06:35:15.919337 6845 raft_consensus.cc:374] T 6d9c9218aa7f454d91786db1d6b07fdc P 4656c42c39664c43be519e461ac32ac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:15.919373 6845 raft_consensus.cc:729] T 6d9c9218aa7f454d91786db1d6b07fdc P 4656c42c39664c43be519e461ac32ac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4656c42c39664c43be519e461ac32ac0, State: Initialized, Role: FOLLOWER
I20260928 06:35:15.919485 6845 consensus_queue.cc:260] T 6d9c9218aa7f454d91786db1d6b07fdc P 4656c42c39664c43be519e461ac32ac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } }
I20260928 06:35:15.919734 6845 ts_tablet_manager.cc:1428] T 6d9c9218aa7f454d91786db1d6b07fdc P 4656c42c39664c43be519e461ac32ac0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:15.919791 6852 tablet_replica.cc:439] T 6d9c9218aa7f454d91786db1d6b07fdc P 4656c42c39664c43be519e461ac32ac0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } } }
I20260928 06:35:15.919941 6852 tablet_replica.cc:442] T 6d9c9218aa7f454d91786db1d6b07fdc P 4656c42c39664c43be519e461ac32ac0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:16.306416 6854 consensus_queue.cc:1035] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:16.310525 6885 consensus_queue.cc:1035] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:16.335860 6885 raft_consensus.cc:482] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:16.336033 6885 raft_consensus.cc:504] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } }
I20260928 06:35:16.336580 6885 leader_election.cc:288] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5241ea335d87483b85f2a0e0c9d727ca (127.5.22.1:39565), 4656c42c39664c43be519e461ac32ac0 (127.5.22.2:36693)
I20260928 06:35:16.337067 6491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d9c9218aa7f454d91786db1d6b07fdc" candidate_uuid: "8016ab6336f343e69863237935cc44e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5241ea335d87483b85f2a0e0c9d727ca" is_pre_election: true
I20260928 06:35:16.337067 6632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d9c9218aa7f454d91786db1d6b07fdc" candidate_uuid: "8016ab6336f343e69863237935cc44e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4656c42c39664c43be519e461ac32ac0" is_pre_election: true
I20260928 06:35:16.337215 6632 raft_consensus.cc:2454] T 6d9c9218aa7f454d91786db1d6b07fdc P 4656c42c39664c43be519e461ac32ac0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8016ab6336f343e69863237935cc44e5 in term 0.
I20260928 06:35:16.337215 6491 raft_consensus.cc:2454] T 6d9c9218aa7f454d91786db1d6b07fdc P 5241ea335d87483b85f2a0e0c9d727ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8016ab6336f343e69863237935cc44e5 in term 0.
I20260928 06:35:16.337517 6706 leader_election.cc:302] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4656c42c39664c43be519e461ac32ac0, 8016ab6336f343e69863237935cc44e5; no voters:
I20260928 06:35:16.337688 6885 raft_consensus.cc:2781] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:16.337749 6885 raft_consensus.cc:482] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:16.337795 6885 raft_consensus.cc:3037] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:16.338891 6885 raft_consensus.cc:504] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } }
I20260928 06:35:16.339239 6885 leader_election.cc:288] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5 [CANDIDATE]: Term 1 election: Requested vote from peers 5241ea335d87483b85f2a0e0c9d727ca (127.5.22.1:39565), 4656c42c39664c43be519e461ac32ac0 (127.5.22.2:36693)
I20260928 06:35:16.339578 6491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d9c9218aa7f454d91786db1d6b07fdc" candidate_uuid: "8016ab6336f343e69863237935cc44e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5241ea335d87483b85f2a0e0c9d727ca"
I20260928 06:35:16.339605 6632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d9c9218aa7f454d91786db1d6b07fdc" candidate_uuid: "8016ab6336f343e69863237935cc44e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4656c42c39664c43be519e461ac32ac0"
I20260928 06:35:16.339673 6491 raft_consensus.cc:3037] T 6d9c9218aa7f454d91786db1d6b07fdc P 5241ea335d87483b85f2a0e0c9d727ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:16.339761 6632 raft_consensus.cc:3037] T 6d9c9218aa7f454d91786db1d6b07fdc P 4656c42c39664c43be519e461ac32ac0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:16.340714 6491 raft_consensus.cc:2454] T 6d9c9218aa7f454d91786db1d6b07fdc P 5241ea335d87483b85f2a0e0c9d727ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8016ab6336f343e69863237935cc44e5 in term 1.
I20260928 06:35:16.340724 6632 raft_consensus.cc:2454] T 6d9c9218aa7f454d91786db1d6b07fdc P 4656c42c39664c43be519e461ac32ac0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8016ab6336f343e69863237935cc44e5 in term 1.
I20260928 06:35:16.340989 6706 leader_election.cc:302] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5241ea335d87483b85f2a0e0c9d727ca, 8016ab6336f343e69863237935cc44e5; no voters:
I20260928 06:35:16.341162 6885 raft_consensus.cc:2781] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:16.341255 6885 raft_consensus.cc:686] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5 [term 1 LEADER]: Becoming Leader. State: Replica: 8016ab6336f343e69863237935cc44e5, State: Running, Role: LEADER
I20260928 06:35:16.341382 6885 consensus_queue.cc:237] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } }
I20260928 06:35:16.342042 6851 tablet_replica.cc:439] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5: TxnStatusTablet state changed. Reason: New leader 8016ab6336f343e69863237935cc44e5. Latest consensus state: current_term: 1 leader_uuid: "8016ab6336f343e69863237935cc44e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } } }
I20260928 06:35:16.342136 6851 tablet_replica.cc:442] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:35:16.342444 6890 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 06:35:16.342599 6890 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 06:35:16.343128 6331 catalog_manager.cc:5495] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8016ab6336f343e69863237935cc44e5 (127.5.22.3). New cstate: current_term: 1 leader_uuid: "8016ab6336f343e69863237935cc44e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } health_report { overall_health: HEALTHY } } }
I20260928 06:35:16.374712 6879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:16.370977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:41125 (local address 127.0.0.1:45374)
0928 06:35:16.371130 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:16.371147 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:16.371275 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:16.371650 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:16.371653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:16.371663 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:35:16.371764 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:16.371769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:16.372781 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:16.372784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:16.373384 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:16.373391 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:16.373995 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:16.373997 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:16.374062 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:16.374106 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:35:16.374435 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:35:16.374451 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 06:35:16.374522 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 06:35:16.374593 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 06:35:16.374761 6893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:16.371052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45374 (local address 127.5.22.3:41125)
0928 06:35:16.371268 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:16.371273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:16.371289 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:35:16.371456 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:16.371459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:16.371513 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:16.371612 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:16.371867 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:16.372672 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:16.373528 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:16.373879 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:16.373981 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:16.374236 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:16.374333 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:16.374406 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:16.374563 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 06:35:16.374615 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":64,"threads_started":1}
I20260928 06:35:16.377558 6632 raft_consensus.cc:1261] T 6d9c9218aa7f454d91786db1d6b07fdc P 4656c42c39664c43be519e461ac32ac0 [term 1 FOLLOWER]: Refusing update from remote peer 8016ab6336f343e69863237935cc44e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:16.377607 6491 raft_consensus.cc:1261] T 6d9c9218aa7f454d91786db1d6b07fdc P 5241ea335d87483b85f2a0e0c9d727ca [term 1 FOLLOWER]: Refusing update from remote peer 8016ab6336f343e69863237935cc44e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:16.377991 6851 consensus_queue.cc:1035] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:16.378144 6885 consensus_queue.cc:1035] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:16.379383 6852 tablet_replica.cc:439] T 6d9c9218aa7f454d91786db1d6b07fdc P 4656c42c39664c43be519e461ac32ac0: TxnStatusTablet state changed. Reason: New leader 8016ab6336f343e69863237935cc44e5. Latest consensus state: current_term: 1 leader_uuid: "8016ab6336f343e69863237935cc44e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } } }
I20260928 06:35:16.379491 6852 tablet_replica.cc:442] T 6d9c9218aa7f454d91786db1d6b07fdc P 4656c42c39664c43be519e461ac32ac0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:16.381026 6885 tablet_replica.cc:439] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8016ab6336f343e69863237935cc44e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } } }
I20260928 06:35:16.381129 6885 tablet_replica.cc:442] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:35:16.381712 6850 tablet_replica.cc:439] T 6d9c9218aa7f454d91786db1d6b07fdc P 5241ea335d87483b85f2a0e0c9d727ca: TxnStatusTablet state changed. Reason: New leader 8016ab6336f343e69863237935cc44e5. Latest consensus state: current_term: 1 leader_uuid: "8016ab6336f343e69863237935cc44e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } } }
I20260928 06:35:16.381835 6850 tablet_replica.cc:442] T 6d9c9218aa7f454d91786db1d6b07fdc P 5241ea335d87483b85f2a0e0c9d727ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:16.381814 6888 mvcc.cc:205] Tried to move back new op lower bound from 7334204687878279168 to 7334204687735570432. Current Snapshot: MvccSnapshot[applied={T|T < 7334204687878279168}]
I20260928 06:35:16.381944 6852 tablet_replica.cc:439] T 6d9c9218aa7f454d91786db1d6b07fdc P 4656c42c39664c43be519e461ac32ac0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8016ab6336f343e69863237935cc44e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } } }
I20260928 06:35:16.382047 6852 tablet_replica.cc:442] T 6d9c9218aa7f454d91786db1d6b07fdc P 4656c42c39664c43be519e461ac32ac0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:16.382014 6884 mvcc.cc:205] Tried to move back new op lower bound from 7334204687878279168 to 7334204687735570432. Current Snapshot: MvccSnapshot[applied={T|T < 7334204687878279168}]
I20260928 06:35:16.382757 6854 tablet_replica.cc:439] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8016ab6336f343e69863237935cc44e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } } }
I20260928 06:35:16.382853 6854 tablet_replica.cc:442] T 6d9c9218aa7f454d91786db1d6b07fdc P 8016ab6336f343e69863237935cc44e5: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:35:16.383569 6886 mvcc.cc:205] Tried to move back new op lower bound from 7334204687878279168 to 7334204687735570432. Current Snapshot: MvccSnapshot[applied={T|T < 7334204687878279168}]
I20260928 06:35:16.383870 6850 tablet_replica.cc:439] T 6d9c9218aa7f454d91786db1d6b07fdc P 5241ea335d87483b85f2a0e0c9d727ca: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8016ab6336f343e69863237935cc44e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241ea335d87483b85f2a0e0c9d727ca" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 39565 } } peers { permanent_uuid: "4656c42c39664c43be519e461ac32ac0" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36693 } } peers { permanent_uuid: "8016ab6336f343e69863237935cc44e5" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 41125 } } }
I20260928 06:35:16.383960 6850 tablet_replica.cc:442] T 6d9c9218aa7f454d91786db1d6b07fdc P 5241ea335d87483b85f2a0e0c9d727ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:16.397835 6893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:16.394252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:52201 (local address 127.5.22.3:41125)
0928 06:35:16.394385 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:16.394390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:16.394636 (+ 246us) server_negotiation.cc:407] Connection header received
0928 06:35:16.394718 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:16.394721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:16.394768 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:16.394845 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:16.395031 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:16.395793 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:16.396663 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:16.397005 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:16.397105 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:16.397378 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:16.397452 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:16.397524 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:16.397631 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 06:35:16.397679 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:35:16.397876 6901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:16.394197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:41125 (local address 127.5.22.3:52201)
0928 06:35:16.394548 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:16.394562 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:16.394649 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:16.394858 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:16.394860 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:16.394865 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:35:16.394948 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:16.394952 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:16.395927 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:16.395930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:16.396543 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:16.396552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:16.397140 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:16.397143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:16.397208 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:16.397248 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:35:16.397544 (+ 296us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:35:16.397550 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:35:16.397619 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:35:16.397683 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":120,"threads_started":1}
I20260928 06:35:16.437589 6332 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:16.437763 6332 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:16.439706 6332 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 179eaae02aec40a48344f40bf270f9d8: Sending DeleteTablet for 3 replicas of tablet fb90c978a1524e7ba5e4ccee37dd790f
I20260928 06:35:16.440208 6471 tablet_service.cc:1514] Processing DeleteTablet for tablet fb90c978a1524e7ba5e4ccee37dd790f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:16 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:45662
I20260928 06:35:16.440312 6612 tablet_service.cc:1514] Processing DeleteTablet for tablet fb90c978a1524e7ba5e4ccee37dd790f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:16 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:36794
I20260928 06:35:16.440590 6905 tablet_replica.cc:330] T fb90c978a1524e7ba5e4ccee37dd790f P 5241ea335d87483b85f2a0e0c9d727ca: stopping tablet replica
I20260928 06:35:16.440596 6754 tablet_service.cc:1514] Processing DeleteTablet for tablet fb90c978a1524e7ba5e4ccee37dd790f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:16 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:45354
I20260928 06:35:16.440706 6905 raft_consensus.cc:2229] T fb90c978a1524e7ba5e4ccee37dd790f P 5241ea335d87483b85f2a0e0c9d727ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:16.440829 6905 raft_consensus.cc:2258] T fb90c978a1524e7ba5e4ccee37dd790f P 5241ea335d87483b85f2a0e0c9d727ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:16.441403 6905 ts_tablet_manager.cc:1905] T fb90c978a1524e7ba5e4ccee37dd790f P 5241ea335d87483b85f2a0e0c9d727ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:16.441880 6907 tablet_replica.cc:330] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5: stopping tablet replica
I20260928 06:35:16.442207 6907 raft_consensus.cc:2229] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:35:16.442441 6907 raft_consensus.cc:2258] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:16.442682 6906 tablet_replica.cc:330] T fb90c978a1524e7ba5e4ccee37dd790f P 4656c42c39664c43be519e461ac32ac0: stopping tablet replica
I20260928 06:35:16.442817 6906 raft_consensus.cc:2229] T fb90c978a1524e7ba5e4ccee37dd790f P 4656c42c39664c43be519e461ac32ac0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:16.442948 6906 raft_consensus.cc:2258] T fb90c978a1524e7ba5e4ccee37dd790f P 4656c42c39664c43be519e461ac32ac0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:16.443454 6907 ts_tablet_manager.cc:1905] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:16.444350 6905 ts_tablet_manager.cc:1918] T fb90c978a1524e7ba5e4ccee37dd790f P 5241ea335d87483b85f2a0e0c9d727ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:35:16.444427 6905 log.cc:1196] T fb90c978a1524e7ba5e4ccee37dd790f P 5241ea335d87483b85f2a0e0c9d727ca: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-0/wal/wals/fb90c978a1524e7ba5e4ccee37dd790f
I20260928 06:35:16.444685 6905 ts_tablet_manager.cc:1939] T fb90c978a1524e7ba5e4ccee37dd790f P 5241ea335d87483b85f2a0e0c9d727ca: Deleting consensus metadata
I20260928 06:35:16.445447 6319 catalog_manager.cc:4841] TS 5241ea335d87483b85f2a0e0c9d727ca (127.5.22.1:39565): tablet fb90c978a1524e7ba5e4ccee37dd790f (table test-table [id=d57cf6274c0e496d80d79e39b398867c]) successfully deleted
I20260928 06:35:16.445807 6906 ts_tablet_manager.cc:1905] T fb90c978a1524e7ba5e4ccee37dd790f P 4656c42c39664c43be519e461ac32ac0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:16.447082 6907 ts_tablet_manager.cc:1918] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:35:16.447216 6907 log.cc:1196] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/fb90c978a1524e7ba5e4ccee37dd790f
I20260928 06:35:16.447525 6907 ts_tablet_manager.cc:1939] T fb90c978a1524e7ba5e4ccee37dd790f P 8016ab6336f343e69863237935cc44e5: Deleting consensus metadata
I20260928 06:35:16.448182 6320 catalog_manager.cc:4841] TS 8016ab6336f343e69863237935cc44e5 (127.5.22.3:41125): tablet fb90c978a1524e7ba5e4ccee37dd790f (table test-table [id=d57cf6274c0e496d80d79e39b398867c]) successfully deleted
I20260928 06:35:16.448628 6906 ts_tablet_manager.cc:1918] T fb90c978a1524e7ba5e4ccee37dd790f P 4656c42c39664c43be519e461ac32ac0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:35:16.448712 6906 log.cc:1196] T fb90c978a1524e7ba5e4ccee37dd790f P 4656c42c39664c43be519e461ac32ac0: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577308481727-5208-0/minicluster-data/ts-1/wal/wals/fb90c978a1524e7ba5e4ccee37dd790f
I20260928 06:35:16.449034 6906 ts_tablet_manager.cc:1939] T fb90c978a1524e7ba5e4ccee37dd790f P 4656c42c39664c43be519e461ac32ac0: Deleting consensus metadata
I20260928 06:35:16.449687 6319 catalog_manager.cc:4841] TS 4656c42c39664c43be519e461ac32ac0 (127.5.22.2:36693): tablet fb90c978a1524e7ba5e4ccee37dd790f (table test-table [id=d57cf6274c0e496d80d79e39b398867c]) successfully deleted
Sep 28 06:35:16 dist-test-slave-j0qf krb5kdc[6288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577315, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.22.1@KRBTEST.COM
I20260928 06:35:16.455969 6909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:16.442017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45664 (local address 127.5.22.1:39565)
0928 06:35:16.444949 (+ 2932us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:16.444954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:16.445171 (+ 217us) server_negotiation.cc:407] Connection header received
0928 06:35:16.445354 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:16.445357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:16.445399 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:16.445455 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:16.446371 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:16.447138 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:16.448337 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:16.448702 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:16.448793 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:16.452332 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:16.452342 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:16.452356 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:16.452390 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:16.454089 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:16.454629 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:16.454635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:16.454636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:16.454719 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:16.455012 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:16.455015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:16.455017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:16.455278 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:16.455508 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:16.455667 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 06:35:16.455790 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2834,"thread_start_us":63,"threads_started":1}
W20260928 06:35:16.456434 6532 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45664
I20260928 06:35:16.465651 6913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:16.458175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46278 (local address 127.5.22.62:33127)
0928 06:35:16.458499 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:16.458503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:16.458516 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:35:16.458558 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:16.458563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:16.458607 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:16.458702 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:16.459397 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:16.459933 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:16.460760 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:16.461024 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:16.461137 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:16.462583 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:16.462587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:16.462590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:16.462615 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:16.463988 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:16.464397 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:16.464401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:16.464403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:16.464491 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:16.464765 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:16.464768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:16.464770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:16.464970 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:16.465114 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:16.465338 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 06:35:16.465464 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":63,"threads_started":1}
W20260928 06:35:16.466064 6332 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46278
I20260928 06:35:16.467232 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 6406
I20260928 06:35:16.475464 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 6547
I20260928 06:35:16.483183 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 6688
I20260928 06:35:16.490478 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 6301
2026-09-28T06:35:16Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3295 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:16 dist-test-slave-j0qf krb5kdc[6916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:16 dist-test-slave-j0qf krb5kdc[6916](info): set up 2 sockets
Sep 28 06:35:16 dist-test-slave-j0qf krb5kdc[6916](info): commencing operation
krb5kdc: starting...
W20260928 06:35:17.533078 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:35:17 dist-test-slave-j0qf krb5kdc[6916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:35:17.657665 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:34005
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:34005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:17.769531 6929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:17.769847 6929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:17.769932 6929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:17.774040 6929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:17.774143 6929 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 06:35:17.774176 6929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:17.774216 6929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:17.774250 6929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:17.774286 6929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:17.780089 6929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:34005
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:34005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:17.781165 6929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:17.782197 6929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:35:17.791838 6929 server_base.cc:1034] running on GCE node
W20260928 06:35:17.791853 6936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:17.791785 6935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:17.791779 6938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:17.792691 6929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:17.793774 6929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:17.794960 6929 hybrid_clock.cc:648] HybridClock initialized: now 1790577317794943 us; error 29 us; skew 500 ppm
Sep 28 06:35:17 dist-test-slave-j0qf krb5kdc[6916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577317, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:17.797715 6929 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:35:17.798707 6929 webserver.cc:460] Webserver started at http://127.5.22.62:45317/ using document root <none> and password file <none>
I20260928 06:35:17.799257 6929 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:17.799347 6929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:17.799582 6929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:17.801605 6929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "bf19afcf2f4149989eb8369a994624f4"
format_stamp: "Formatted at 2026-09-28 06:35:17 on dist-test-slave-j0qf"
server_key: "fe907c43021d7502baa09019f59b152c"
server_key_iv: "9a72c8486aeed27ef778733935257c09"
server_key_version: "encryptionkey@0"
I20260928 06:35:17.802202 6929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "bf19afcf2f4149989eb8369a994624f4"
format_stamp: "Formatted at 2026-09-28 06:35:17 on dist-test-slave-j0qf"
server_key: "fe907c43021d7502baa09019f59b152c"
server_key_iv: "9a72c8486aeed27ef778733935257c09"
server_key_version: "encryptionkey@0"
I20260928 06:35:17.806002 6929 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:17.808471 6945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:17.809726 6929 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:17.809877 6929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "bf19afcf2f4149989eb8369a994624f4"
format_stamp: "Formatted at 2026-09-28 06:35:17 on dist-test-slave-j0qf"
server_key: "fe907c43021d7502baa09019f59b152c"
server_key_iv: "9a72c8486aeed27ef778733935257c09"
server_key_version: "encryptionkey@0"
I20260928 06:35:17.810022 6929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:17.829252 6929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:17.833050 6929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:17.833290 6929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:17.841735 6929 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:34005
I20260928 06:35:17.841812 7007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:34005 every 8 connection(s)
I20260928 06:35:17.842811 6929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:17.844878 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 6929
I20260928 06:35:17.845009 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:35:17.845258 5208 external_mini_cluster.cc:1428] Setting key d4ba566928375f28908aba33dfb13f06
I20260928 06:35:17.845741 7008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:17.853085 7008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4: Bootstrap starting.
Sep 28 06:35:17 dist-test-slave-j0qf krb5kdc[6916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:17.855718 7008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:17.856528 7008 log.cc:824] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:17.858595 7008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4: No bootstrap required, opened a new log
I20260928 06:35:17.861542 7011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:17.846864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55158 (local address 127.5.22.62:34005)
0928 06:35:17.847309 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:17.847331 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:17.847375 (+ 44us) server_negotiation.cc:407] Connection header received
0928 06:35:17.848141 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:17.848165 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:17.848607 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:17.848988 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:17.849700 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:17.851206 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:17.852142 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:17.852819 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:17.852986 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:17.855977 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:17.855983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:17.855997 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:17.856028 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:17.858561 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:17.859036 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:17.859043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:17.859047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:17.859164 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:17.859547 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:17.859553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:17.859554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:17.859998 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:17.860201 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:17.860501 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 06:35:17.860801 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":123,"threads_started":1}
I20260928 06:35:17.861819 7008 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf19afcf2f4149989eb8369a994624f4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34005 } }
I20260928 06:35:17.862066 7008 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:17.862159 7008 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf19afcf2f4149989eb8369a994624f4, State: Initialized, Role: FOLLOWER
I20260928 06:35:17.862754 7008 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf19afcf2f4149989eb8369a994624f4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34005 } }
I20260928 06:35:17.862920 7008 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:17.862993 7008 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:17.863081 7008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:17.864197 7008 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf19afcf2f4149989eb8369a994624f4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34005 } }
I20260928 06:35:17.864490 7008 leader_election.cc:302] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf19afcf2f4149989eb8369a994624f4; no voters:
I20260928 06:35:17.864758 7008 leader_election.cc:288] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:17.864912 7013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:17.865140 7013 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4 [term 1 LEADER]: Becoming Leader. State: Replica: bf19afcf2f4149989eb8369a994624f4, State: Running, Role: LEADER
I20260928 06:35:17.865479 7013 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf19afcf2f4149989eb8369a994624f4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34005 } }
I20260928 06:35:17.866029 7008 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:17.867429 7014 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf19afcf2f4149989eb8369a994624f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf19afcf2f4149989eb8369a994624f4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34005 } } }
I20260928 06:35:17.867558 7014 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:17.867761 7015 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf19afcf2f4149989eb8369a994624f4. Latest consensus state: current_term: 1 leader_uuid: "bf19afcf2f4149989eb8369a994624f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf19afcf2f4149989eb8369a994624f4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34005 } } }
I20260928 06:35:17.868109 7015 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:17.868451 7024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:17.871244 7024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:17.876838 7024 catalog_manager.cc:1347] Generated new cluster ID: a1aa9943af24481e91ec28b979e65caa
I20260928 06:35:17.876909 7024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:17.892599 7024 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:17.893529 7024 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:17.901916 7024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4: Generated new TSK 0
I20260928 06:35:17.902833 7024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.22.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
I20260928 06:35:17.941289 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:34005
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:18.054943 7034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:18.055225 7034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:18.055274 7034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:18.059662 7034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:18.059755 7034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:18.059834 7034 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 06:35:18.059919 7034 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:35:18.060019 7034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:35:18.065356 7034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.22.62:34005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:18.066510 7034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:18.067672 7034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:35:18.077670 7034 server_base.cc:1034] running on GCE node
W20260928 06:35:18.077641 7042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:18.077656 7040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:18.077864 7039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:18.078262 7034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:18.078987 7034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:18.080195 7034 hybrid_clock.cc:648] HybridClock initialized: now 1790577318080178 us; error 30 us; skew 500 ppm
Sep 28 06:35:18 dist-test-slave-j0qf krb5kdc[6916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577318, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:18.083029 7034 init.cc:377] Logged in from keytab as kudu/127.5.22.1@KRBTEST.COM (short username kudu)
I20260928 06:35:18.083985 7034 webserver.cc:460] Webserver started at http://127.5.22.1:35115/ using document root <none> and password file <none>
I20260928 06:35:18.084576 7034 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:18.084669 7034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:18.084901 7034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:18.087011 7034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "4f742b414fa349beada5febca578eda1"
format_stamp: "Formatted at 2026-09-28 06:35:18 on dist-test-slave-j0qf"
server_key: "875af191a85a04bff15652ccc30ea5b9"
server_key_iv: "d53ce3c58a7291f40c0997999a6abf52"
server_key_version: "encryptionkey@0"
I20260928 06:35:18.087586 7034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "4f742b414fa349beada5febca578eda1"
format_stamp: "Formatted at 2026-09-28 06:35:18 on dist-test-slave-j0qf"
server_key: "875af191a85a04bff15652ccc30ea5b9"
server_key_iv: "d53ce3c58a7291f40c0997999a6abf52"
server_key_version: "encryptionkey@0"
I20260928 06:35:18.091333 7034 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 06:35:18.093865 7049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:18.095144 7034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:35:18.095284 7034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "4f742b414fa349beada5febca578eda1"
format_stamp: "Formatted at 2026-09-28 06:35:18 on dist-test-slave-j0qf"
server_key: "875af191a85a04bff15652ccc30ea5b9"
server_key_iv: "d53ce3c58a7291f40c0997999a6abf52"
server_key_version: "encryptionkey@0"
I20260928 06:35:18.095395 7034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:18.112859 7034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:18.116144 7034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:18.116357 7034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:18.117866 7034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:18.117937 7034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:18.118011 7034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:18.118057 7034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:35:18 dist-test-slave-j0qf krb5kdc[6916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577318, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:18.129482 7034 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:37213
I20260928 06:35:18.129621 7165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:37213 every 8 connection(s)
I20260928 06:35:18.130535 7034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:18.132336 7011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.118787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:52505 (local address 127.5.22.62:34005)
0928 06:35:18.118925 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:18.118929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:18.119968 (+ 1039us) server_negotiation.cc:407] Connection header received
0928 06:35:18.121082 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:18.121086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:18.121156 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:18.121266 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:18.122779 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.123722 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.124905 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.125149 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.125273 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.128450 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:18.128454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:18.128457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:18.128486 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:18.130194 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:18.130804 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:18.130807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:18.130808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:18.130890 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:18.131328 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:18.131331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:18.131333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:18.131537 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:18.131648 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:18.132105 (+ 457us) server_negotiation.cc:299] Negotiation successful
0928 06:35:18.132214 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:35:18.133441 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.119334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:34005 (local address 127.5.22.1:52505)
0928 06:35:18.119812 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:18.119856 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:18.120802 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:18.121475 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:18.121482 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:18.121981 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:18.122613 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:18.122636 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.123867 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.123871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:18.124769 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:18.124778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.125333 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.125337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:18.125522 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.126166 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:18.126186 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:18.128293 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:18.130311 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:18.130334 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:18.130348 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:18.130703 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:18.131036 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:18.131039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:18.131041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:18.131212 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:18.131672 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:18.131677 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:18.131890 (+ 213us) client_negotiation.cc:769] Sending connection context
0928 06:35:18.132038 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 06:35:18.132279 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":136,"threads_started":1}
I20260928 06:35:18.134872 7166 heartbeater.cc:347] Connected to a master server at 127.5.22.62:34005
I20260928 06:35:18.135020 7166 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:18.135522 7166 heartbeater.cc:511] Master 127.5.22.62:34005 requested a full tablet report, sending...
I20260928 06:35:18.137120 6961 ts_manager.cc:194] Registered new tserver with Master: 4f742b414fa349beada5febca578eda1 (127.5.22.1:37213)
I20260928 06:35:18.137984 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 7034
I20260928 06:35:18.138134 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:18.138504 5208 external_mini_cluster.cc:1428] Setting key ad70dbbb82702e95db7c78e6e9248f93
I20260928 06:35:18.138732 6961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.1@KRBTEST.COM'} at 127.5.22.1:52505
I20260928 06:35:18.145205 7011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.137224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:48451 (local address 127.5.22.62:34005)
0928 06:35:18.137340 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:18.137344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:18.137592 (+ 248us) server_negotiation.cc:407] Connection header received
0928 06:35:18.137732 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:18.137735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:18.137787 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:18.137867 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:18.138380 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.139043 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.140199 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.140501 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.140616 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.141946 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:18.141948 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:18.141950 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:18.141972 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:18.143447 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:18.143929 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:18.143931 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:18.143932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:18.144002 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:18.144300 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:18.144302 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:18.144303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:18.144456 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:18.144530 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:18.144987 (+ 457us) server_negotiation.cc:299] Negotiation successful
0928 06:35:18.145097 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:35:18.145221 7172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.137141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:34005 (local address 127.5.22.1:48451)
0928 06:35:18.137498 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:18.137512 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:18.137621 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:18.137918 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:18.137921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:18.138132 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:18.138247 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:18.138254 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.139215 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.139218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:18.140071 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:18.140081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.140642 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.140646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:18.140754 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.141288 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:18.141309 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:18.141804 (+ 495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:18.143548 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:18.143554 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:18.143564 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:18.143830 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:18.144092 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:18.144098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:18.144103 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:18.144207 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:18.144556 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:18.144570 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:18.144686 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 06:35:18.144811 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 06:35:18.144983 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":134,"threads_started":1}
WARNING: no policy specified for kudu/127.5.22.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
I20260928 06:35:18.169935 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:34005
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:18.283352 7175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:18.283665 7175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:18.283744 7175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:18.287863 7175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:18.287956 7175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:18.288026 7175 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 06:35:18.288079 7175 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:35:18.288147 7175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:35:18.293463 7175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.22.62:34005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:18.294559 7175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:18.295639 7175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:18.305346 7181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:18.305388 7180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:18.305388 7183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:18.305423 7175 server_base.cc:1034] running on GCE node
I20260928 06:35:18.305968 7175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:18.306701 7175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:18.307863 7175 hybrid_clock.cc:648] HybridClock initialized: now 1790577318307849 us; error 29 us; skew 500 ppm
Sep 28 06:35:18 dist-test-slave-j0qf krb5kdc[6916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577318, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:18.310706 7175 init.cc:377] Logged in from keytab as kudu/127.5.22.2@KRBTEST.COM (short username kudu)
I20260928 06:35:18.311611 7175 webserver.cc:460] Webserver started at http://127.5.22.2:34765/ using document root <none> and password file <none>
I20260928 06:35:18.312134 7175 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:18.312194 7175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:18.312357 7175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:18.314597 7175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "796dc2765a354d0b84b066a5813b79fc"
format_stamp: "Formatted at 2026-09-28 06:35:18 on dist-test-slave-j0qf"
server_key: "9f8b2571349b35e7cd0ce48101673c35"
server_key_iv: "14a4d47545dd325c4c1ed64265077c24"
server_key_version: "encryptionkey@0"
I20260928 06:35:18.315178 7175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "796dc2765a354d0b84b066a5813b79fc"
format_stamp: "Formatted at 2026-09-28 06:35:18 on dist-test-slave-j0qf"
server_key: "9f8b2571349b35e7cd0ce48101673c35"
server_key_iv: "14a4d47545dd325c4c1ed64265077c24"
server_key_version: "encryptionkey@0"
I20260928 06:35:18.318832 7175 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:18.321173 7190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:18.322520 7175 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:35:18.322671 7175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "796dc2765a354d0b84b066a5813b79fc"
format_stamp: "Formatted at 2026-09-28 06:35:18 on dist-test-slave-j0qf"
server_key: "9f8b2571349b35e7cd0ce48101673c35"
server_key_iv: "14a4d47545dd325c4c1ed64265077c24"
server_key_version: "encryptionkey@0"
I20260928 06:35:18.322765 7175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:18.340613 7175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:18.343575 7175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:18.343772 7175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:18.345319 7175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:18.345368 7175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:18.345408 7175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:18.345463 7175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:35:18 dist-test-slave-j0qf krb5kdc[6916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577318, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:18.356576 7175 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:35601
I20260928 06:35:18.356694 7310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:35601 every 8 connection(s)
I20260928 06:35:18.357688 7175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:18.359455 7011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.346112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:60509 (local address 127.5.22.62:34005)
0928 06:35:18.346269 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:18.346274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:18.347232 (+ 958us) server_negotiation.cc:407] Connection header received
0928 06:35:18.348219 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:18.348223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:18.348295 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:18.348346 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:18.349737 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.350418 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.351653 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.351875 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.351990 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.355549 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:18.355553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:18.355555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:18.355592 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:18.357273 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:18.357868 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:18.357872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:18.357874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:18.357959 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:18.358392 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:18.358395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:18.358408 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:18.358710 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:18.358793 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:18.359227 (+ 434us) server_negotiation.cc:299] Negotiation successful
0928 06:35:18.359337 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 06:35:18.360287 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.346603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:34005 (local address 127.5.22.2:60509)
0928 06:35:18.347061 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:18.347111 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:18.347946 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:18.348756 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:18.348763 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:18.349098 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:18.349576 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:18.349595 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.350561 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.350564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:18.351524 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:18.351533 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.352045 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.352049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:18.352206 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.352926 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:18.352952 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:18.355371 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:18.357386 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:18.357390 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:18.357402 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:18.357762 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:18.358059 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:18.358062 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:18.358064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:18.358246 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:18.358790 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:18.358795 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:18.359016 (+ 221us) client_negotiation.cc:769] Sending connection context
0928 06:35:18.359178 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 06:35:18.359426 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":121,"threads_started":1}
I20260928 06:35:18.366549 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 7175
I20260928 06:35:18.366662 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:18.366915 5208 external_mini_cluster.cc:1428] Setting key b5a10f5b1eb11fcde726ceab2b4d161f
I20260928 06:35:18.367403 7311 heartbeater.cc:347] Connected to a master server at 127.5.22.62:34005
I20260928 06:35:18.367586 7311 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:18.368117 7311 heartbeater.cc:511] Master 127.5.22.62:34005 requested a full tablet report, sending...
I20260928 06:35:18.369410 6961 ts_manager.cc:194] Registered new tserver with Master: 796dc2765a354d0b84b066a5813b79fc (127.5.22.2:35601)
I20260928 06:35:18.370060 6961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.2@KRBTEST.COM'} at 127.5.22.2:60509
I20260928 06:35:18.377552 7317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.368259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:34005 (local address 127.5.22.2:56285)
0928 06:35:18.368627 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:18.368641 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:18.368769 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:18.369108 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:18.369111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:18.370468 (+ 1357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:18.370572 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:18.370578 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.371352 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.371355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:18.372298 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:18.372309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.372771 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.372776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:18.372888 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.373447 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:18.373458 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:18.373823 (+ 365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:18.376058 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:18.376062 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:18.376064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:18.376257 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:18.376571 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:18.376573 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:18.376574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:18.376647 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:18.377143 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:18.377145 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:18.377218 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 06:35:18.377302 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 06:35:18.377407 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":87,"threads_started":1}
I20260928 06:35:18.377722 7011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.368326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:56285 (local address 127.5.22.62:34005)
0928 06:35:18.368447 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:18.368451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:18.368703 (+ 252us) server_negotiation.cc:407] Connection header received
0928 06:35:18.368868 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:18.368871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:18.368915 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:18.368962 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:18.370722 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.371225 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.372435 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.372653 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.372737 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.373981 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:18.373987 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:18.373993 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:18.374024 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:18.375941 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:18.376357 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:18.376363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:18.376365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:18.376463 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:18.376738 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:18.376744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:18.376747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:18.377039 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:18.377156 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:18.377385 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 06:35:18.377517 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.5.22.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
I20260928 06:35:18.397605 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:34005
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:18.517776 7320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:18.518075 7320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:18.518126 7320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:18.522145 7320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:18.522217 7320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:18.522258 7320 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 06:35:18.522274 7320 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:35:18.522305 7320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:35:18.527966 7320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.22.62:34005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:18.529027 7320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:18.530033 7320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:18.539929 7329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:18.539915 7327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:18.539921 7326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:18.540215 7320 server_base.cc:1034] running on GCE node
I20260928 06:35:18.540736 7320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:18.541442 7320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:18.542701 7320 hybrid_clock.cc:648] HybridClock initialized: now 1790577318542686 us; error 29 us; skew 500 ppm
Sep 28 06:35:18 dist-test-slave-j0qf krb5kdc[6916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577318, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:18.545610 7320 init.cc:377] Logged in from keytab as kudu/127.5.22.3@KRBTEST.COM (short username kudu)
I20260928 06:35:18.546532 7320 webserver.cc:460] Webserver started at http://127.5.22.3:41275/ using document root <none> and password file <none>
I20260928 06:35:18.547091 7320 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:18.547179 7320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:18.547422 7320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:18.549544 7320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad"
format_stamp: "Formatted at 2026-09-28 06:35:18 on dist-test-slave-j0qf"
server_key: "a374c4dca79f314af0d0011340ebed4f"
server_key_iv: "8ee413c165bb8b6e0e35f52f2a60790b"
server_key_version: "encryptionkey@0"
I20260928 06:35:18.550148 7320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad"
format_stamp: "Formatted at 2026-09-28 06:35:18 on dist-test-slave-j0qf"
server_key: "a374c4dca79f314af0d0011340ebed4f"
server_key_iv: "8ee413c165bb8b6e0e35f52f2a60790b"
server_key_version: "encryptionkey@0"
I20260928 06:35:18.553885 7320 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:35:18.556268 7336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:18.557508 7320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:18.557647 7320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad"
format_stamp: "Formatted at 2026-09-28 06:35:18 on dist-test-slave-j0qf"
server_key: "a374c4dca79f314af0d0011340ebed4f"
server_key_iv: "8ee413c165bb8b6e0e35f52f2a60790b"
server_key_version: "encryptionkey@0"
I20260928 06:35:18.557765 7320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:18.578948 7320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:18.581946 7320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:18.582155 7320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:18.583678 7320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:18.583751 7320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:18.583811 7320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:18.583861 7320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:35:18 dist-test-slave-j0qf krb5kdc[6916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577318, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:18.595016 7320 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:40259
I20260928 06:35:18.596005 7320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:18.596863 7453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:40259 every 8 connection(s)
I20260928 06:35:18.597666 7011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.584591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:37661 (local address 127.5.22.62:34005)
0928 06:35:18.584744 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:18.584749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:18.585641 (+ 892us) server_negotiation.cc:407] Connection header received
0928 06:35:18.586808 (+ 1167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:18.586815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:18.586888 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:18.586998 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:18.588158 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.588750 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.590056 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.590277 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.590429 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.593870 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:18.593874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:18.593877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:18.593912 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:18.595498 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:18.596091 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:18.596096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:18.596101 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:18.596186 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:18.596598 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:18.596601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:18.596603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:18.596860 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:18.596969 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:18.597439 (+ 470us) server_negotiation.cc:299] Negotiation successful
0928 06:35:18.597540 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 06:35:18.598404 7345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.585012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:34005 (local address 127.5.22.3:37661)
0928 06:35:18.585458 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:18.585506 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:18.586554 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:18.587184 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:18.587191 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:18.587559 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:18.588013 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:18.588025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.588870 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.588876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:18.589921 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:18.589930 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.590478 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.590482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:18.590694 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.591314 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:18.591329 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:18.593678 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:18.595617 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:18.595626 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:18.595642 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:18.595964 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:18.596276 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:18.596282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:18.596287 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:18.596478 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:18.596974 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:18.596980 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:18.597223 (+ 243us) client_negotiation.cc:769] Sending connection context
0928 06:35:18.597379 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 06:35:18.597618 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":130,"threads_started":1}
I20260928 06:35:18.599630 7454 heartbeater.cc:347] Connected to a master server at 127.5.22.62:34005
I20260928 06:35:18.599778 7454 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:18.600404 7454 heartbeater.cc:511] Master 127.5.22.62:34005 requested a full tablet report, sending...
I20260928 06:35:18.601438 6961 ts_manager.cc:194] Registered new tserver with Master: 7ecd8bc05f2747b8ae7e7a3bbe2f8dad (127.5.22.3:40259)
I20260928 06:35:18.602180 6961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.3@KRBTEST.COM'} at 127.5.22.3:37661
I20260928 06:35:18.604697 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 7320
I20260928 06:35:18.604812 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:18.605085 5208 external_mini_cluster.cc:1428] Setting key 895eeef68db51b60dafa2b396ac1c765
I20260928 06:35:18.607079 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:35:18.609587 7011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.601898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:59985 (local address 127.5.22.62:34005)
0928 06:35:18.602021 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:18.602025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:18.602233 (+ 208us) server_negotiation.cc:407] Connection header received
0928 06:35:18.602400 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:18.602403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:18.602444 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:18.602533 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:18.603069 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.603551 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.604577 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.604816 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.604937 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.606406 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:18.606408 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:18.606410 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:18.606431 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:18.607900 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:18.608336 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:18.608344 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:18.608346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:18.608415 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:18.608799 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:18.608802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:18.608803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:18.608960 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:18.609077 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:18.609317 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 06:35:18.609420 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:35:18.609644 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.601821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:34005 (local address 127.5.22.3:59985)
0928 06:35:18.602149 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:18.602160 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:18.602296 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:18.602584 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:18.602588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:18.602808 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:18.602946 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:18.602954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.603680 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.603683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:18.604460 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:18.604474 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.605035 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.605037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:18.605157 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.605696 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:18.605710 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:18.606150 (+ 440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:18.608022 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:18.608027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:18.608029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:18.608224 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:18.608508 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:18.608512 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:18.608513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:18.608606 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:18.609061 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:18.609064 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:18.609151 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 06:35:18.609265 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 06:35:18.609409 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":4,"thread_start_us":107,"threads_started":1}
Sep 28 06:35:18 dist-test-slave-j0qf krb5kdc[6916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:35:18 dist-test-slave-j0qf krb5kdc[6916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:18.633061 7011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.624609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59668 (local address 127.5.22.62:34005)
0928 06:35:18.624743 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:18.624747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:18.624760 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:35:18.624801 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:18.624804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:18.624842 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:18.624917 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:18.625576 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.626174 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.627167 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.627436 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.627517 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.629844 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:18.629847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:18.629849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:18.629873 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:18.631316 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:18.631759 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:18.631762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:18.631764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:18.631846 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:18.632126 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:18.632131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:18.632133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:18.632313 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:18.632473 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:18.632710 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 06:35:18.632834 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 06:35:18.636145 6961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:35:18.639104 6961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:18.653687 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.647336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54856 (local address 127.5.22.1:37213)
0928 06:35:18.648612 (+ 1276us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:18.648618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:18.648639 (+ 21us) server_negotiation.cc:407] Connection header received
0928 06:35:18.648699 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:18.648702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:18.648848 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:18.649003 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:18.649317 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.650541 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.652076 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.652806 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.652922 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.653073 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:18.653439 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 06:35:18.653539 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1159,"thread_start_us":82,"threads_started":1}
I20260928 06:35:18.653707 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.647193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:37213 (local address 127.0.0.1:54856)
0928 06:35:18.648249 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:18.648262 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:18.648345 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:18.649005 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:18.649011 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:18.649035 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:18.649186 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:18.649193 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.650654 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.650657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:18.651954 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:18.651968 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.652965 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.652969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:18.653189 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.653261 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 06:35:18.653384 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 06:35:18.653535 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":162,"thread_start_us":74,"threads_started":1}
I20260928 06:35:18.653741 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.647422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46848 (local address 127.5.22.3:40259)
0928 06:35:18.648140 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:18.648152 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:18.648176 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:35:18.648660 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:18.648668 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:18.648873 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:18.649027 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:18.649435 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.650157 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.651947 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.652806 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.652922 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.653095 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:18.653525 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 06:35:18.653618 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":153,"threads_started":1}
I20260928 06:35:18.654247 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.647193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:40259 (local address 127.0.0.1:46848)
0928 06:35:18.648032 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:18.648086 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:18.648460 (+ 374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:18.649200 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:18.649202 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:18.649221 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:18.649334 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:18.649339 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.650271 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.650278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:18.651817 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:18.651827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.652965 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.652969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:18.653184 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.653261 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 06:35:18.653384 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 06:35:18.654104 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":1387,"thread_start_us":86,"threads_started":1}
I20260928 06:35:18.654731 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.647842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44002 (local address 127.5.22.2:35601)
0928 06:35:18.648403 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:18.648417 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:18.648443 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:35:18.648632 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:18.648638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:18.648849 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:18.649003 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:18.649473 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.650618 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.653161 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.653935 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.654038 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.654465 (+ 427us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:18.654511 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 06:35:18.654602 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":128,"threads_started":1}
I20260928 06:35:18.655130 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.647444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:35601 (local address 127.0.0.1:44002)
0928 06:35:18.647998 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:18.648061 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:18.648189 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:18.649173 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:18.649175 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:18.649190 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:18.649338 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:18.649347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.650710 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.650713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:18.651826 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:18.651836 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.654075 (+ 2239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.654078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:18.654172 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.654195 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 06:35:18.654262 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:35:18.654368 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":26,"spinlock_wait_cycles":1792,"thread_start_us":96,"threads_started":1}
I20260928 06:35:18.656908 7387 tablet_service.cc:1467] Processing CreateTablet for tablet 5450415360b94f7c939d0c7c69b9aa02 (DEFAULT_TABLE table=test-table [id=981999ee374c4699b60c61351ed3be5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:18.657006 7100 tablet_service.cc:1467] Processing CreateTablet for tablet 5450415360b94f7c939d0c7c69b9aa02 (DEFAULT_TABLE table=test-table [id=981999ee374c4699b60c61351ed3be5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:18.657800 7387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5450415360b94f7c939d0c7c69b9aa02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:18.657476 7241 tablet_service.cc:1467] Processing CreateTablet for tablet 5450415360b94f7c939d0c7c69b9aa02 (DEFAULT_TABLE table=test-table [id=981999ee374c4699b60c61351ed3be5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:18.657991 7100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5450415360b94f7c939d0c7c69b9aa02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:18.658305 7241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5450415360b94f7c939d0c7c69b9aa02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:18.664958 7477 tablet_bootstrap.cc:492] T 5450415360b94f7c939d0c7c69b9aa02 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Bootstrap starting.
I20260928 06:35:18.666235 7478 tablet_bootstrap.cc:492] T 5450415360b94f7c939d0c7c69b9aa02 P 4f742b414fa349beada5febca578eda1: Bootstrap starting.
I20260928 06:35:18.666306 7479 tablet_bootstrap.cc:492] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc: Bootstrap starting.
I20260928 06:35:18.667289 7477 tablet_bootstrap.cc:654] T 5450415360b94f7c939d0c7c69b9aa02 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:18.668064 7477 log.cc:824] T 5450415360b94f7c939d0c7c69b9aa02 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:18.668434 7478 tablet_bootstrap.cc:654] T 5450415360b94f7c939d0c7c69b9aa02 P 4f742b414fa349beada5febca578eda1: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:18.668434 7479 tablet_bootstrap.cc:654] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:18.669262 7479 log.cc:824] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:18.669262 7478 log.cc:824] T 5450415360b94f7c939d0c7c69b9aa02 P 4f742b414fa349beada5febca578eda1: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:18.671123 7478 tablet_bootstrap.cc:492] T 5450415360b94f7c939d0c7c69b9aa02 P 4f742b414fa349beada5febca578eda1: No bootstrap required, opened a new log
I20260928 06:35:18.671239 7477 tablet_bootstrap.cc:492] T 5450415360b94f7c939d0c7c69b9aa02 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: No bootstrap required, opened a new log
I20260928 06:35:18.671238 7479 tablet_bootstrap.cc:492] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc: No bootstrap required, opened a new log
I20260928 06:35:18.671336 7478 ts_tablet_manager.cc:1397] T 5450415360b94f7c939d0c7c69b9aa02 P 4f742b414fa349beada5febca578eda1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 06:35:18.671419 7479 ts_tablet_manager.cc:1397] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 06:35:18.671419 7477 ts_tablet_manager.cc:1397] T 5450415360b94f7c939d0c7c69b9aa02 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 06:35:18.674902 7478 raft_consensus.cc:348] T 5450415360b94f7c939d0c7c69b9aa02 P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } }
I20260928 06:35:18.675158 7478 raft_consensus.cc:374] T 5450415360b94f7c939d0c7c69b9aa02 P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:18.675428 7478 raft_consensus.cc:729] T 5450415360b94f7c939d0c7c69b9aa02 P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f742b414fa349beada5febca578eda1, State: Initialized, Role: FOLLOWER
I20260928 06:35:18.675350 7477 raft_consensus.cc:348] T 5450415360b94f7c939d0c7c69b9aa02 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } }
I20260928 06:35:18.675350 7479 raft_consensus.cc:348] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } }
I20260928 06:35:18.675596 7479 raft_consensus.cc:374] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:18.675596 7477 raft_consensus.cc:374] T 5450415360b94f7c939d0c7c69b9aa02 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:18.675647 7479 raft_consensus.cc:729] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 796dc2765a354d0b84b066a5813b79fc, State: Initialized, Role: FOLLOWER
I20260928 06:35:18.675745 7477 raft_consensus.cc:729] T 5450415360b94f7c939d0c7c69b9aa02 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ecd8bc05f2747b8ae7e7a3bbe2f8dad, State: Initialized, Role: FOLLOWER
I20260928 06:35:18.675918 7478 consensus_queue.cc:260] T 5450415360b94f7c939d0c7c69b9aa02 P 4f742b414fa349beada5febca578eda1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } }
I20260928 06:35:18.676131 7477 consensus_queue.cc:260] T 5450415360b94f7c939d0c7c69b9aa02 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } }
I20260928 06:35:18.676110 7479 consensus_queue.cc:260] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } }
I20260928 06:35:18.676800 7454 heartbeater.cc:503] Master 127.5.22.62:34005 was elected leader, sending a full tablet report...
I20260928 06:35:18.677060 7479 ts_tablet_manager.cc:1428] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 06:35:18.677078 7311 heartbeater.cc:503] Master 127.5.22.62:34005 was elected leader, sending a full tablet report...
I20260928 06:35:18.678025 7478 ts_tablet_manager.cc:1428] T 5450415360b94f7c939d0c7c69b9aa02 P 4f742b414fa349beada5febca578eda1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 06:35:18.678045 7477 ts_tablet_manager.cc:1428] T 5450415360b94f7c939d0c7c69b9aa02 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 06:35:18.678375 7166 heartbeater.cc:503] Master 127.5.22.62:34005 was elected leader, sending a full tablet report...
I20260928 06:35:18.823621 7484 raft_consensus.cc:482] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:18.823875 7484 raft_consensus.cc:504] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } }
I20260928 06:35:18.825040 7484 leader_election.cc:288] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f742b414fa349beada5febca578eda1 (127.5.22.1:37213), 7ecd8bc05f2747b8ae7e7a3bbe2f8dad (127.5.22.3:40259)
I20260928 06:35:18.829183 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.825430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:35959 (local address 127.5.22.3:40259)
0928 06:35:18.825556 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:18.825574 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:18.825622 (+ 48us) server_negotiation.cc:407] Connection header received
0928 06:35:18.825813 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:18.825817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:18.825877 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:18.825953 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:18.826294 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.826949 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.828037 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.828630 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.828745 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.828812 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:18.829015 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 06:35:18.829075 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:35:18.829209 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.825334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:40259 (local address 127.5.22.2:35959)
0928 06:35:18.825538 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:18.825555 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:18.825668 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:18.825988 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:18.825991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:18.826009 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:18.826159 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:18.826167 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.827083 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.827088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:18.827901 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:18.827910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.828773 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.828776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:18.828883 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.828909 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 06:35:18.828993 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 06:35:18.829075 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260928 06:35:18.829912 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.825427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:42947 (local address 127.5.22.1:37213)
0928 06:35:18.825663 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:18.825668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:18.825799 (+ 131us) server_negotiation.cc:407] Connection header received
0928 06:35:18.825925 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:18.825949 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:18.826004 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:18.826075 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:18.826427 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.827033 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.828755 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.829353 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.829459 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.829515 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:18.829760 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 06:35:18.829821 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260928 06:35:18.829952 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.825338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:37213 (local address 127.5.22.2:42947)
0928 06:35:18.825655 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:18.825676 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:18.825817 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:18.826112 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:18.826114 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:18.826128 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:18.826263 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:18.826271 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.827146 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.827148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:18.828618 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:18.828629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.829482 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.829485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:18.829630 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.829655 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 06:35:18.829737 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 06:35:18.829802 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":28,"thread_start_us":67,"threads_started":1}
I20260928 06:35:18.829939 7407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5450415360b94f7c939d0c7c69b9aa02" candidate_uuid: "796dc2765a354d0b84b066a5813b79fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" is_pre_election: true
I20260928 06:35:18.830298 7407 raft_consensus.cc:2454] T 5450415360b94f7c939d0c7c69b9aa02 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 796dc2765a354d0b84b066a5813b79fc in term 0.
I20260928 06:35:18.830762 7120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5450415360b94f7c939d0c7c69b9aa02" candidate_uuid: "796dc2765a354d0b84b066a5813b79fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f742b414fa349beada5febca578eda1" is_pre_election: true
I20260928 06:35:18.830849 7192 leader_election.cc:302] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 796dc2765a354d0b84b066a5813b79fc, 7ecd8bc05f2747b8ae7e7a3bbe2f8dad; no voters:
I20260928 06:35:18.831058 7120 raft_consensus.cc:2454] T 5450415360b94f7c939d0c7c69b9aa02 P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 796dc2765a354d0b84b066a5813b79fc in term 0.
I20260928 06:35:18.831154 7484 raft_consensus.cc:2781] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:18.831239 7484 raft_consensus.cc:482] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:18.831305 7484 raft_consensus.cc:3037] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:18.832463 7484 raft_consensus.cc:504] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } }
I20260928 06:35:18.832827 7484 leader_election.cc:288] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc [CANDIDATE]: Term 1 election: Requested vote from peers 4f742b414fa349beada5febca578eda1 (127.5.22.1:37213), 7ecd8bc05f2747b8ae7e7a3bbe2f8dad (127.5.22.3:40259)
I20260928 06:35:18.833197 7407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5450415360b94f7c939d0c7c69b9aa02" candidate_uuid: "796dc2765a354d0b84b066a5813b79fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad"
I20260928 06:35:18.833227 7120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5450415360b94f7c939d0c7c69b9aa02" candidate_uuid: "796dc2765a354d0b84b066a5813b79fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f742b414fa349beada5febca578eda1"
I20260928 06:35:18.833343 7407 raft_consensus.cc:3037] T 5450415360b94f7c939d0c7c69b9aa02 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:18.833382 7120 raft_consensus.cc:3037] T 5450415360b94f7c939d0c7c69b9aa02 P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:18.834576 7120 raft_consensus.cc:2454] T 5450415360b94f7c939d0c7c69b9aa02 P 4f742b414fa349beada5febca578eda1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 796dc2765a354d0b84b066a5813b79fc in term 1.
I20260928 06:35:18.834915 7193 leader_election.cc:302] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f742b414fa349beada5febca578eda1, 796dc2765a354d0b84b066a5813b79fc; no voters:
I20260928 06:35:18.835084 7407 raft_consensus.cc:2454] T 5450415360b94f7c939d0c7c69b9aa02 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 796dc2765a354d0b84b066a5813b79fc in term 1.
I20260928 06:35:18.835109 7484 raft_consensus.cc:2781] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:18.835353 7484 raft_consensus.cc:686] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc [term 1 LEADER]: Becoming Leader. State: Replica: 796dc2765a354d0b84b066a5813b79fc, State: Running, Role: LEADER
I20260928 06:35:18.835701 7484 consensus_queue.cc:237] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } }
I20260928 06:35:18.838861 6961 catalog_manager.cc:5495] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 796dc2765a354d0b84b066a5813b79fc (127.5.22.2). New cstate: current_term: 1 leader_uuid: "796dc2765a354d0b84b066a5813b79fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } health_report { overall_health: HEALTHY } } }
W20260928 06:35:18.847086 7455 tablet.cc:2369] T 5450415360b94f7c939d0c7c69b9aa02 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:35:18.859018 7312 tablet.cc:2369] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:35:18.882074 7167 tablet.cc:2369] T 5450415360b94f7c939d0c7c69b9aa02 P 4f742b414fa349beada5febca578eda1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 06:35:18 dist-test-slave-j0qf krb5kdc[6916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577317, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:18.909049 7011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.900398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59680 (local address 127.5.22.62:34005)
0928 06:35:18.900522 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:18.900526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:18.900818 (+ 292us) server_negotiation.cc:407] Connection header received
0928 06:35:18.900978 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:18.900982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:18.901027 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:18.901132 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:18.901792 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.902371 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.903174 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.903394 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.903504 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.905912 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:18.905915 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:18.905918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:18.905937 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:18.907335 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:18.907798 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:18.907804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:18.907806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:18.907886 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:18.908153 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:18.908159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:18.908161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:18.908336 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:18.908442 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:18.908742 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 06:35:18.908911 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":47}
I20260928 06:35:18.909049 7494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.900391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:34005 (local address 127.0.0.1:59680)
0928 06:35:18.900729 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:18.900742 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:18.900880 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:18.901156 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:18.901159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:18.901586 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:18.901674 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:18.901681 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.902478 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.902484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:18.903056 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:18.903064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.903509 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.903512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:18.903606 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.904246 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:18.904259 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:18.905744 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:18.907445 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:18.907457 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:18.907463 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:18.907699 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:18.907983 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:18.907985 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:18.907987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:18.908066 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:18.908442 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:18.908450 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:18.908572 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 06:35:18.908688 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 06:35:18.908848 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":119,"threads_started":1}
I20260928 06:35:18.914062 6961 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:59680:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 06:35:18.914952 6961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:18.919963 7100 tablet_service.cc:1467] Processing CreateTablet for tablet 8c1deaee7590493e901e8d925f11aeca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b7dea8505e8541e3a2078eced384a1e6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:35:18.920044 7241 tablet_service.cc:1467] Processing CreateTablet for tablet 8c1deaee7590493e901e8d925f11aeca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b7dea8505e8541e3a2078eced384a1e6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:35:18.920168 7387 tablet_service.cc:1467] Processing CreateTablet for tablet 8c1deaee7590493e901e8d925f11aeca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b7dea8505e8541e3a2078eced384a1e6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:35:18.920733 7241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c1deaee7590493e901e8d925f11aeca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:18.920740 7100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c1deaee7590493e901e8d925f11aeca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:18.920861 7387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c1deaee7590493e901e8d925f11aeca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:18.923744 7479 tablet_bootstrap.cc:492] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc: Bootstrap starting.
I20260928 06:35:18.923775 7477 tablet_bootstrap.cc:492] T 8c1deaee7590493e901e8d925f11aeca P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Bootstrap starting.
I20260928 06:35:18.924588 7479 tablet_bootstrap.cc:654] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:18.924649 7477 tablet_bootstrap.cc:654] T 8c1deaee7590493e901e8d925f11aeca P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:18.925302 7478 tablet_bootstrap.cc:492] T 8c1deaee7590493e901e8d925f11aeca P 4f742b414fa349beada5febca578eda1: Bootstrap starting.
I20260928 06:35:18.925868 7479 tablet_bootstrap.cc:492] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc: No bootstrap required, opened a new log
I20260928 06:35:18.925868 7477 tablet_bootstrap.cc:492] T 8c1deaee7590493e901e8d925f11aeca P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: No bootstrap required, opened a new log
I20260928 06:35:18.925943 7479 ts_tablet_manager.cc:1397] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:35:18.925943 7477 ts_tablet_manager.cc:1397] T 8c1deaee7590493e901e8d925f11aeca P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:35:18.926252 7478 tablet_bootstrap.cc:654] T 8c1deaee7590493e901e8d925f11aeca P 4f742b414fa349beada5febca578eda1: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:18.926465 7477 raft_consensus.cc:348] T 8c1deaee7590493e901e8d925f11aeca P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:18.926503 7479 raft_consensus.cc:348] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:18.926594 7479 raft_consensus.cc:374] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:18.926594 7477 raft_consensus.cc:374] T 8c1deaee7590493e901e8d925f11aeca P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:18.926627 7479 raft_consensus.cc:729] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 796dc2765a354d0b84b066a5813b79fc, State: Initialized, Role: FOLLOWER
I20260928 06:35:18.926627 7477 raft_consensus.cc:729] T 8c1deaee7590493e901e8d925f11aeca P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ecd8bc05f2747b8ae7e7a3bbe2f8dad, State: Initialized, Role: FOLLOWER
I20260928 06:35:18.926744 7477 consensus_queue.cc:260] T 8c1deaee7590493e901e8d925f11aeca P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:18.926744 7479 consensus_queue.cc:260] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:18.927006 7477 ts_tablet_manager.cc:1428] T 8c1deaee7590493e901e8d925f11aeca P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:18.927006 7479 ts_tablet_manager.cc:1428] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:18.927109 7485 tablet_replica.cc:439] T 8c1deaee7590493e901e8d925f11aeca P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } } }
I20260928 06:35:18.927250 7485 tablet_replica.cc:442] T 8c1deaee7590493e901e8d925f11aeca P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:18.927484 7484 tablet_replica.cc:439] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } } }
I20260928 06:35:18.927646 7484 tablet_replica.cc:442] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:18.928406 7478 tablet_bootstrap.cc:492] T 8c1deaee7590493e901e8d925f11aeca P 4f742b414fa349beada5febca578eda1: No bootstrap required, opened a new log
I20260928 06:35:18.928493 7478 ts_tablet_manager.cc:1397] T 8c1deaee7590493e901e8d925f11aeca P 4f742b414fa349beada5febca578eda1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 06:35:18.928964 7478 raft_consensus.cc:348] T 8c1deaee7590493e901e8d925f11aeca P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:18.929104 7478 raft_consensus.cc:374] T 8c1deaee7590493e901e8d925f11aeca P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:18.929143 7478 raft_consensus.cc:729] T 8c1deaee7590493e901e8d925f11aeca P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f742b414fa349beada5febca578eda1, State: Initialized, Role: FOLLOWER
I20260928 06:35:18.929268 7478 consensus_queue.cc:260] T 8c1deaee7590493e901e8d925f11aeca P 4f742b414fa349beada5febca578eda1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:18.929535 7478 ts_tablet_manager.cc:1428] T 8c1deaee7590493e901e8d925f11aeca P 4f742b414fa349beada5febca578eda1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:18.929772 7483 tablet_replica.cc:439] T 8c1deaee7590493e901e8d925f11aeca P 4f742b414fa349beada5febca578eda1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } } }
I20260928 06:35:18.929873 7483 tablet_replica.cc:442] T 8c1deaee7590493e901e8d925f11aeca P 4f742b414fa349beada5febca578eda1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:19.255869 7484 raft_consensus.cc:482] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:19.256021 7484 raft_consensus.cc:504] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:19.256515 7484 leader_election.cc:288] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f742b414fa349beada5febca578eda1 (127.5.22.1:37213), 7ecd8bc05f2747b8ae7e7a3bbe2f8dad (127.5.22.3:40259)
I20260928 06:35:19.257077 7120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1deaee7590493e901e8d925f11aeca" candidate_uuid: "796dc2765a354d0b84b066a5813b79fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f742b414fa349beada5febca578eda1" is_pre_election: true
I20260928 06:35:19.257074 7407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1deaee7590493e901e8d925f11aeca" candidate_uuid: "796dc2765a354d0b84b066a5813b79fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" is_pre_election: true
I20260928 06:35:19.257239 7120 raft_consensus.cc:2454] T 8c1deaee7590493e901e8d925f11aeca P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 796dc2765a354d0b84b066a5813b79fc in term 0.
I20260928 06:35:19.257254 7407 raft_consensus.cc:2454] T 8c1deaee7590493e901e8d925f11aeca P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 796dc2765a354d0b84b066a5813b79fc in term 0.
I20260928 06:35:19.257557 7193 leader_election.cc:302] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f742b414fa349beada5febca578eda1, 796dc2765a354d0b84b066a5813b79fc; no voters:
I20260928 06:35:19.257738 7484 raft_consensus.cc:2781] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:19.257804 7484 raft_consensus.cc:482] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:19.257843 7484 raft_consensus.cc:3037] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:19.258931 7484 raft_consensus.cc:504] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:19.259277 7484 leader_election.cc:288] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc [CANDIDATE]: Term 1 election: Requested vote from peers 4f742b414fa349beada5febca578eda1 (127.5.22.1:37213), 7ecd8bc05f2747b8ae7e7a3bbe2f8dad (127.5.22.3:40259)
I20260928 06:35:19.259650 7120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1deaee7590493e901e8d925f11aeca" candidate_uuid: "796dc2765a354d0b84b066a5813b79fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f742b414fa349beada5febca578eda1"
I20260928 06:35:19.259778 7120 raft_consensus.cc:3037] T 8c1deaee7590493e901e8d925f11aeca P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:19.259734 7407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1deaee7590493e901e8d925f11aeca" candidate_uuid: "796dc2765a354d0b84b066a5813b79fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad"
I20260928 06:35:19.259878 7407 raft_consensus.cc:3037] T 8c1deaee7590493e901e8d925f11aeca P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:19.260730 7120 raft_consensus.cc:2454] T 8c1deaee7590493e901e8d925f11aeca P 4f742b414fa349beada5febca578eda1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 796dc2765a354d0b84b066a5813b79fc in term 1.
I20260928 06:35:19.260869 7407 raft_consensus.cc:2454] T 8c1deaee7590493e901e8d925f11aeca P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 796dc2765a354d0b84b066a5813b79fc in term 1.
I20260928 06:35:19.261046 7193 leader_election.cc:302] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f742b414fa349beada5febca578eda1, 796dc2765a354d0b84b066a5813b79fc; no voters:
I20260928 06:35:19.261312 7484 raft_consensus.cc:2781] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:19.261422 7484 raft_consensus.cc:686] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc [term 1 LEADER]: Becoming Leader. State: Replica: 796dc2765a354d0b84b066a5813b79fc, State: Running, Role: LEADER
I20260928 06:35:19.261576 7484 consensus_queue.cc:237] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:19.262274 7487 tablet_replica.cc:439] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc: TxnStatusTablet state changed. Reason: New leader 796dc2765a354d0b84b066a5813b79fc. Latest consensus state: current_term: 1 leader_uuid: "796dc2765a354d0b84b066a5813b79fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } } }
I20260928 06:35:19.262439 7487 tablet_replica.cc:442] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:35:19.262701 7499 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 06:35:19.262969 7499 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 06:35:19.263384 6962 catalog_manager.cc:5495] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 796dc2765a354d0b84b066a5813b79fc (127.5.22.2). New cstate: current_term: 1 leader_uuid: "796dc2765a354d0b84b066a5813b79fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } health_report { overall_health: UNKNOWN } } }
I20260928 06:35:19.274084 7484 consensus_queue.cc:1035] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:19.280017 7487 consensus_queue.cc:1035] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:19.292703 7507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:19.288535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44004 (local address 127.5.22.2:35601)
0928 06:35:19.288832 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:19.288838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:19.288851 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:35:19.288907 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:19.288909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:19.288967 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:19.289070 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:19.289336 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.289869 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.291139 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.291390 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.291542 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:19.291906 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:19.292316 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:19.292487 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:19.292516 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 06:35:19.292564 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":133,"threads_started":1}
I20260928 06:35:19.292738 7494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:19.288379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:35601 (local address 127.0.0.1:44004)
0928 06:35:19.288623 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:19.288639 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:19.288736 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:19.289118 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:19.289122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:19.289134 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:35:19.289246 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:19.289252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.289974 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.289979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:19.290998 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:19.291010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.291566 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.291570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:19.291667 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:19.291725 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:35:19.292424 (+ 699us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:35:19.292440 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 06:35:19.292501 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:35:19.292590 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260928 06:35:19.297688 7407 raft_consensus.cc:1261] T 8c1deaee7590493e901e8d925f11aeca P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 1 FOLLOWER]: Refusing update from remote peer 796dc2765a354d0b84b066a5813b79fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:19.297686 7120 raft_consensus.cc:1261] T 8c1deaee7590493e901e8d925f11aeca P 4f742b414fa349beada5febca578eda1 [term 1 FOLLOWER]: Refusing update from remote peer 796dc2765a354d0b84b066a5813b79fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:19.298206 7484 consensus_queue.cc:1035] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:19.298426 7487 consensus_queue.cc:1035] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:19.301134 7485 tablet_replica.cc:439] T 8c1deaee7590493e901e8d925f11aeca P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: TxnStatusTablet state changed. Reason: New leader 796dc2765a354d0b84b066a5813b79fc. Latest consensus state: current_term: 1 leader_uuid: "796dc2765a354d0b84b066a5813b79fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } } }
I20260928 06:35:19.301255 7485 tablet_replica.cc:442] T 8c1deaee7590493e901e8d925f11aeca P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:19.301566 7483 tablet_replica.cc:439] T 8c1deaee7590493e901e8d925f11aeca P 4f742b414fa349beada5febca578eda1: TxnStatusTablet state changed. Reason: New leader 796dc2765a354d0b84b066a5813b79fc. Latest consensus state: current_term: 1 leader_uuid: "796dc2765a354d0b84b066a5813b79fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } } }
I20260928 06:35:19.301697 7483 tablet_replica.cc:442] T 8c1deaee7590493e901e8d925f11aeca P 4f742b414fa349beada5febca578eda1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:19.305037 7484 tablet_replica.cc:439] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "796dc2765a354d0b84b066a5813b79fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } } }
I20260928 06:35:19.305153 7484 tablet_replica.cc:442] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:35:19.305312 7487 tablet_replica.cc:439] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "796dc2765a354d0b84b066a5813b79fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } } }
I20260928 06:35:19.305415 7487 tablet_replica.cc:442] T 8c1deaee7590493e901e8d925f11aeca P 796dc2765a354d0b84b066a5813b79fc: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:35:19.305634 7483 tablet_replica.cc:439] T 8c1deaee7590493e901e8d925f11aeca P 4f742b414fa349beada5febca578eda1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "796dc2765a354d0b84b066a5813b79fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } } }
I20260928 06:35:19.305733 7483 tablet_replica.cc:442] T 8c1deaee7590493e901e8d925f11aeca P 4f742b414fa349beada5febca578eda1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:19.306710 7485 tablet_replica.cc:439] T 8c1deaee7590493e901e8d925f11aeca P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "796dc2765a354d0b84b066a5813b79fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } } }
I20260928 06:35:19.306826 7485 tablet_replica.cc:442] T 8c1deaee7590493e901e8d925f11aeca P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:19.308025 7503 mvcc.cc:205] Tried to move back new op lower bound from 7334204699838316544 to 7334204699696824320. Current Snapshot: MvccSnapshot[applied={T|T < 7334204699838316544}]
I20260928 06:35:19.309044 7505 mvcc.cc:205] Tried to move back new op lower bound from 7334204699838316544 to 7334204699696824320. Current Snapshot: MvccSnapshot[applied={T|T < 7334204699838316544}]
I20260928 06:35:19.316717 7507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:19.313069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44012 (local address 127.5.22.2:35601)
0928 06:35:19.313207 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:19.313213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:19.313500 (+ 287us) server_negotiation.cc:407] Connection header received
0928 06:35:19.313676 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:19.313680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:19.313731 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:19.313828 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:19.314121 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.314732 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.315624 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.315862 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.315976 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:19.316260 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:19.316337 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:19.316412 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:19.316519 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 06:35:19.316571 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:35:19.328274 7507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:19.324656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:33693 (local address 127.5.22.2:35601)
0928 06:35:19.324784 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:19.324788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:19.325010 (+ 222us) server_negotiation.cc:407] Connection header received
0928 06:35:19.325138 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:19.325140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:19.325178 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:19.325241 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:19.325519 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.326030 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.327103 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.327374 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.327463 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:19.327819 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:19.327887 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:19.327976 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:19.328058 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 06:35:19.328106 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":49}
I20260928 06:35:19.328281 7518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:19.324605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:35601 (local address 127.5.22.2:33693)
0928 06:35:19.324919 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:19.324935 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:19.325043 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:19.325273 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:19.325276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:19.325296 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:35:19.325414 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:19.325420 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.326174 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.326177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:19.326980 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:19.326989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.327546 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.327549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:19.327632 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:19.327669 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:35:19.327978 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:35:19.327985 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 06:35:19.328050 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:35:19.328121 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":120,"threads_started":1}
I20260928 06:35:19.368531 6962 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:19.368717 6962 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:19.371016 6962 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4: Sending DeleteTablet for 3 replicas of tablet 5450415360b94f7c939d0c7c69b9aa02
I20260928 06:35:19.371706 7100 tablet_service.cc:1514] Processing DeleteTablet for tablet 5450415360b94f7c939d0c7c69b9aa02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:19 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:54856
I20260928 06:35:19.372078 7387 tablet_service.cc:1514] Processing DeleteTablet for tablet 5450415360b94f7c939d0c7c69b9aa02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:19 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:46848
I20260928 06:35:19.372494 7521 tablet_replica.cc:330] T 5450415360b94f7c939d0c7c69b9aa02 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: stopping tablet replica
I20260928 06:35:19.372799 7521 raft_consensus.cc:2229] T 5450415360b94f7c939d0c7c69b9aa02 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:19.373011 7521 raft_consensus.cc:2258] T 5450415360b94f7c939d0c7c69b9aa02 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:19.373486 6962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:35:19.373893 6962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:19.374455 7521 ts_tablet_manager.cc:1905] T 5450415360b94f7c939d0c7c69b9aa02 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:19.375918 7241 tablet_service.cc:1514] Processing DeleteTablet for tablet 5450415360b94f7c939d0c7c69b9aa02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:19 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:44002
I20260928 06:35:19.376441 7522 tablet_replica.cc:330] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc: stopping tablet replica
I20260928 06:35:19.376773 7522 raft_consensus.cc:2229] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:35:19.377184 7522 raft_consensus.cc:2258] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:19.378049 7521 ts_tablet_manager.cc:1918] T 5450415360b94f7c939d0c7c69b9aa02 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:35:19.378149 7521 log.cc:1196] T 5450415360b94f7c939d0c7c69b9aa02 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/5450415360b94f7c939d0c7c69b9aa02
I20260928 06:35:19.378479 7521 ts_tablet_manager.cc:1939] T 5450415360b94f7c939d0c7c69b9aa02 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Deleting consensus metadata
I20260928 06:35:19.379585 7100 tablet_service.cc:1467] Processing CreateTablet for tablet 93e9d5dfce844201a5c673326c88d50e (DEFAULT_TABLE table=test-table [id=2a2f7ba1631f4bcba607c48dce7fcf1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:19.379737 7241 tablet_service.cc:1467] Processing CreateTablet for tablet 93e9d5dfce844201a5c673326c88d50e (DEFAULT_TABLE table=test-table [id=2a2f7ba1631f4bcba607c48dce7fcf1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:19.379774 6947 catalog_manager.cc:4841] TS 7ecd8bc05f2747b8ae7e7a3bbe2f8dad (127.5.22.3:40259): tablet 5450415360b94f7c939d0c7c69b9aa02 (table test-table [id=981999ee374c4699b60c61351ed3be5e]) successfully deleted
I20260928 06:35:19.379855 7100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93e9d5dfce844201a5c673326c88d50e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:19.379959 7241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93e9d5dfce844201a5c673326c88d50e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:19.380414 7522 ts_tablet_manager.cc:1905] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:19.382360 7387 tablet_service.cc:1467] Processing CreateTablet for tablet 93e9d5dfce844201a5c673326c88d50e (DEFAULT_TABLE table=test-table [id=2a2f7ba1631f4bcba607c48dce7fcf1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:19.382598 7387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93e9d5dfce844201a5c673326c88d50e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:19.383047 7522 ts_tablet_manager.cc:1918] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:35:19.383122 7522 log.cc:1196] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-1/wal/wals/5450415360b94f7c939d0c7c69b9aa02
I20260928 06:35:19.383395 7522 ts_tablet_manager.cc:1939] T 5450415360b94f7c939d0c7c69b9aa02 P 796dc2765a354d0b84b066a5813b79fc: Deleting consensus metadata
I20260928 06:35:19.384464 7477 tablet_bootstrap.cc:492] T 93e9d5dfce844201a5c673326c88d50e P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Bootstrap starting.
I20260928 06:35:19.384465 7520 tablet_replica.cc:330] T 5450415360b94f7c939d0c7c69b9aa02 P 4f742b414fa349beada5febca578eda1: stopping tablet replica
I20260928 06:35:19.384579 7478 tablet_bootstrap.cc:492] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1: Bootstrap starting.
I20260928 06:35:19.384759 7520 raft_consensus.cc:2229] T 5450415360b94f7c939d0c7c69b9aa02 P 4f742b414fa349beada5febca578eda1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:19.384945 7520 raft_consensus.cc:2258] T 5450415360b94f7c939d0c7c69b9aa02 P 4f742b414fa349beada5febca578eda1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:19.385401 7477 tablet_bootstrap.cc:654] T 93e9d5dfce844201a5c673326c88d50e P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:19.385455 7478 tablet_bootstrap.cc:654] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:19.386027 7479 tablet_bootstrap.cc:492] T 93e9d5dfce844201a5c673326c88d50e P 796dc2765a354d0b84b066a5813b79fc: Bootstrap starting.
I20260928 06:35:19.386163 7520 ts_tablet_manager.cc:1905] T 5450415360b94f7c939d0c7c69b9aa02 P 4f742b414fa349beada5febca578eda1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:19.387526 6946 catalog_manager.cc:4841] TS 796dc2765a354d0b84b066a5813b79fc (127.5.22.2:35601): tablet 5450415360b94f7c939d0c7c69b9aa02 (table test-table [id=981999ee374c4699b60c61351ed3be5e]) successfully deleted
I20260928 06:35:19.387969 7479 tablet_bootstrap.cc:654] T 93e9d5dfce844201a5c673326c88d50e P 796dc2765a354d0b84b066a5813b79fc: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:19.388774 7520 ts_tablet_manager.cc:1918] T 5450415360b94f7c939d0c7c69b9aa02 P 4f742b414fa349beada5febca578eda1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:35:19.388868 7520 log.cc:1196] T 5450415360b94f7c939d0c7c69b9aa02 P 4f742b414fa349beada5febca578eda1: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-0/wal/wals/5450415360b94f7c939d0c7c69b9aa02
I20260928 06:35:19.388978 7479 tablet_bootstrap.cc:492] T 93e9d5dfce844201a5c673326c88d50e P 796dc2765a354d0b84b066a5813b79fc: No bootstrap required, opened a new log
I20260928 06:35:19.389055 7479 ts_tablet_manager.cc:1397] T 93e9d5dfce844201a5c673326c88d50e P 796dc2765a354d0b84b066a5813b79fc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 06:35:19.389168 7520 ts_tablet_manager.cc:1939] T 5450415360b94f7c939d0c7c69b9aa02 P 4f742b414fa349beada5febca578eda1: Deleting consensus metadata
I20260928 06:35:19.389437 7479 raft_consensus.cc:348] T 93e9d5dfce844201a5c673326c88d50e P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:19.389559 7479 raft_consensus.cc:374] T 93e9d5dfce844201a5c673326c88d50e P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:19.389598 7479 raft_consensus.cc:729] T 93e9d5dfce844201a5c673326c88d50e P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 796dc2765a354d0b84b066a5813b79fc, State: Initialized, Role: FOLLOWER
I20260928 06:35:19.389724 7479 consensus_queue.cc:260] T 93e9d5dfce844201a5c673326c88d50e P 796dc2765a354d0b84b066a5813b79fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:19.390005 7479 ts_tablet_manager.cc:1428] T 93e9d5dfce844201a5c673326c88d50e P 796dc2765a354d0b84b066a5813b79fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:19.390261 6948 catalog_manager.cc:4841] TS 4f742b414fa349beada5febca578eda1 (127.5.22.1:37213): tablet 5450415360b94f7c939d0c7c69b9aa02 (table test-table [id=981999ee374c4699b60c61351ed3be5e]) successfully deleted
I20260928 06:35:19.390997 7478 tablet_bootstrap.cc:492] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1: No bootstrap required, opened a new log
I20260928 06:35:19.391075 7478 ts_tablet_manager.cc:1397] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260928 06:35:19.391144 7477 tablet_bootstrap.cc:492] T 93e9d5dfce844201a5c673326c88d50e P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: No bootstrap required, opened a new log
I20260928 06:35:19.391220 7477 ts_tablet_manager.cc:1397] T 93e9d5dfce844201a5c673326c88d50e P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260928 06:35:19.391507 7478 raft_consensus.cc:348] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:19.391628 7478 raft_consensus.cc:374] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:19.391662 7478 raft_consensus.cc:729] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f742b414fa349beada5febca578eda1, State: Initialized, Role: FOLLOWER
I20260928 06:35:19.391662 7477 raft_consensus.cc:348] T 93e9d5dfce844201a5c673326c88d50e P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:19.391783 7477 raft_consensus.cc:374] T 93e9d5dfce844201a5c673326c88d50e P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:19.391815 7477 raft_consensus.cc:729] T 93e9d5dfce844201a5c673326c88d50e P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ecd8bc05f2747b8ae7e7a3bbe2f8dad, State: Initialized, Role: FOLLOWER
I20260928 06:35:19.391784 7478 consensus_queue.cc:260] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:19.391938 7477 consensus_queue.cc:260] T 93e9d5dfce844201a5c673326c88d50e P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:19.392056 7478 ts_tablet_manager.cc:1428] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:19.392192 7477 ts_tablet_manager.cc:1428] T 93e9d5dfce844201a5c673326c88d50e P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:19.476908 7483 raft_consensus.cc:482] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:19.477110 7483 raft_consensus.cc:504] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:19.478451 7483 leader_election.cc:288] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 796dc2765a354d0b84b066a5813b79fc (127.5.22.2:35601), 7ecd8bc05f2747b8ae7e7a3bbe2f8dad (127.5.22.3:40259)
I20260928 06:35:19.483850 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:19.478718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:40259 (local address 127.5.22.1:44933)
0928 06:35:19.479094 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:19.479110 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:19.479239 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:19.479734 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:19.479737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:19.479756 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:19.479923 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:19.479930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.481109 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.481112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:19.482199 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:19.482211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.483324 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.483328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:19.483439 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:19.483473 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 06:35:19.483554 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 06:35:19.483642 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":27,"spinlock_wait_cycles":32256,"thread_start_us":111,"threads_started":1}
I20260928 06:35:19.484069 7507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:19.478752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:45169 (local address 127.5.22.2:35601)
0928 06:35:19.478896 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:19.478901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:19.479160 (+ 259us) server_negotiation.cc:407] Connection header received
0928 06:35:19.479424 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:19.479428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:19.479481 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:19.479583 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:19.480005 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.480895 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.482797 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.483417 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.483529 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:19.483601 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:19.483856 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 06:35:19.483941 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:35:19.484084 7526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:19.478653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:35601 (local address 127.5.22.1:45169)
0928 06:35:19.479053 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:19.479070 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:19.479239 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:19.479609 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:19.479613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:19.479634 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:19.479851 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:19.479859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.481072 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.481076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:19.482653 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:19.482664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.483534 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.483540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:19.483716 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:19.483741 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 06:35:19.483814 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:35:19.483899 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":77,"spinlock_wait_cycles":2304,"thread_start_us":100,"threads_started":1}
I20260928 06:35:19.484560 7528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:19.479120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:44933 (local address 127.5.22.3:40259)
0928 06:35:19.479480 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:19.479489 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:19.479501 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:35:19.479574 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:19.479576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:19.479626 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:19.479722 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:19.480043 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.480934 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.482371 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.483153 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.483313 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:19.483388 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:19.484287 (+ 899us) server_negotiation.cc:299] Negotiation successful
0928 06:35:19.484390 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":77,"threads_started":1}
I20260928 06:35:19.484789 7262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e9d5dfce844201a5c673326c88d50e" candidate_uuid: "4f742b414fa349beada5febca578eda1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "796dc2765a354d0b84b066a5813b79fc" is_pre_election: true
I20260928 06:35:19.485047 7262 raft_consensus.cc:2454] T 93e9d5dfce844201a5c673326c88d50e P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f742b414fa349beada5febca578eda1 in term 0.
I20260928 06:35:19.485072 7407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e9d5dfce844201a5c673326c88d50e" candidate_uuid: "4f742b414fa349beada5febca578eda1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" is_pre_election: true
I20260928 06:35:19.485237 7407 raft_consensus.cc:2454] T 93e9d5dfce844201a5c673326c88d50e P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f742b414fa349beada5febca578eda1 in term 0.
I20260928 06:35:19.485514 7050 leader_election.cc:302] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f742b414fa349beada5febca578eda1, 796dc2765a354d0b84b066a5813b79fc; no voters:
I20260928 06:35:19.485823 7483 raft_consensus.cc:2781] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:19.485882 7483 raft_consensus.cc:482] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:19.485924 7483 raft_consensus.cc:3037] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:19.487067 7483 raft_consensus.cc:504] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:19.487505 7483 leader_election.cc:288] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1 [CANDIDATE]: Term 1 election: Requested vote from peers 796dc2765a354d0b84b066a5813b79fc (127.5.22.2:35601), 7ecd8bc05f2747b8ae7e7a3bbe2f8dad (127.5.22.3:40259)
I20260928 06:35:19.487895 7262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e9d5dfce844201a5c673326c88d50e" candidate_uuid: "4f742b414fa349beada5febca578eda1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "796dc2765a354d0b84b066a5813b79fc"
I20260928 06:35:19.487907 7407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e9d5dfce844201a5c673326c88d50e" candidate_uuid: "4f742b414fa349beada5febca578eda1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad"
I20260928 06:35:19.488036 7407 raft_consensus.cc:3037] T 93e9d5dfce844201a5c673326c88d50e P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:19.488046 7262 raft_consensus.cc:3037] T 93e9d5dfce844201a5c673326c88d50e P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:19.489017 7262 raft_consensus.cc:2454] T 93e9d5dfce844201a5c673326c88d50e P 796dc2765a354d0b84b066a5813b79fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f742b414fa349beada5febca578eda1 in term 1.
I20260928 06:35:19.489157 7407 raft_consensus.cc:2454] T 93e9d5dfce844201a5c673326c88d50e P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f742b414fa349beada5febca578eda1 in term 1.
I20260928 06:35:19.489383 7050 leader_election.cc:302] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f742b414fa349beada5febca578eda1, 796dc2765a354d0b84b066a5813b79fc; no voters:
I20260928 06:35:19.489570 7483 raft_consensus.cc:2781] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:19.489853 7483 raft_consensus.cc:686] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1 [term 1 LEADER]: Becoming Leader. State: Replica: 4f742b414fa349beada5febca578eda1, State: Running, Role: LEADER
I20260928 06:35:19.490135 7483 consensus_queue.cc:237] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:19.492802 6961 catalog_manager.cc:5495] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f742b414fa349beada5febca578eda1 (127.5.22.1). New cstate: current_term: 1 leader_uuid: "4f742b414fa349beada5febca578eda1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } health_report { overall_health: UNKNOWN } } }
I20260928 06:35:19.520655 7531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:19.516810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54864 (local address 127.5.22.1:37213)
0928 06:35:19.517154 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:19.517161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:19.517172 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:35:19.517238 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:19.517240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:19.517290 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:19.517419 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:19.517670 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.518223 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.519133 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.519377 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.519495 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:19.519776 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:19.520237 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:19.520370 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:19.520430 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 06:35:19.520486 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":130,"threads_started":1}
I20260928 06:35:19.532112 7534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:19.528072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:35601 (local address 127.5.22.1:42255)
0928 06:35:19.528388 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:19.528400 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:19.528481 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:19.528743 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:19.528745 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:19.528750 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:35:19.528850 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:19.528853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.529833 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.529835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:19.530863 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:19.530872 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.531418 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.531421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:19.531491 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:19.531540 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:35:19.531842 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:35:19.531847 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:35:19.531897 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:35:19.531963 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":114,"threads_started":1}
I20260928 06:35:19.532102 7507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:19.528191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:42255 (local address 127.5.22.2:35601)
0928 06:35:19.528395 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:19.528399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:19.528447 (+ 48us) server_negotiation.cc:407] Connection header received
0928 06:35:19.528591 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:19.528594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:19.528648 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:19.528728 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:19.528951 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.529726 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.530972 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.531310 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.531410 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:19.531679 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:19.531756 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:19.531828 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:19.531926 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 06:35:19.531976 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260928 06:35:19.538538 7407 raft_consensus.cc:1261] T 93e9d5dfce844201a5c673326c88d50e P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 1 FOLLOWER]: Refusing update from remote peer 4f742b414fa349beada5febca578eda1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:19.538756 7262 raft_consensus.cc:1261] T 93e9d5dfce844201a5c673326c88d50e P 796dc2765a354d0b84b066a5813b79fc [term 1 FOLLOWER]: Refusing update from remote peer 4f742b414fa349beada5febca578eda1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:19.538995 7529 consensus_queue.cc:1035] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:19.539225 7483 consensus_queue.cc:1035] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:19.546933 7502 mvcc.cc:205] Tried to move back new op lower bound from 7334204700825395200 to 7334204700634054656. Current Snapshot: MvccSnapshot[applied={T|T < 7334204700825395200}]
I20260928 06:35:19.557169 7531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:19.554071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:33329 (local address 127.5.22.1:37213)
0928 06:35:19.554164 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:19.554168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:19.554182 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:35:19.554222 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:19.554225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:19.554274 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:19.554382 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:19.554652 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.555159 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.555983 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.556303 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.556403 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:19.556736 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:19.556813 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:19.556890 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:19.556983 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 06:35:19.557040 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:35:19.557191 7518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:19.553295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:37213 (local address 127.5.22.2:33329)
0928 06:35:19.553538 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:19.553557 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:19.553631 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:19.554406 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:19.554410 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:19.554421 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:35:19.554532 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:19.554538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.555268 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.555274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:19.555861 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:19.555869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.556509 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.556515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:19.556588 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:19.556613 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:35:19.556906 (+ 293us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:35:19.556913 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 06:35:19.556973 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:35:19.557083 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
W20260928 06:35:19.557868 7503 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 06:35:21.121543 6961 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:21.121838 6961 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:21.124595 6961 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4: Sending DeleteTablet for 3 replicas of tablet 93e9d5dfce844201a5c673326c88d50e
I20260928 06:35:21.125211 7100 tablet_service.cc:1514] Processing DeleteTablet for tablet 93e9d5dfce844201a5c673326c88d50e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:21 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:54856
I20260928 06:35:21.125216 7241 tablet_service.cc:1514] Processing DeleteTablet for tablet 93e9d5dfce844201a5c673326c88d50e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:21 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:44002
I20260928 06:35:21.125483 7387 tablet_service.cc:1514] Processing DeleteTablet for tablet 93e9d5dfce844201a5c673326c88d50e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:21 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:46848
I20260928 06:35:21.125634 7562 tablet_replica.cc:330] T 93e9d5dfce844201a5c673326c88d50e P 796dc2765a354d0b84b066a5813b79fc: stopping tablet replica
I20260928 06:35:21.125775 7562 raft_consensus.cc:2229] T 93e9d5dfce844201a5c673326c88d50e P 796dc2765a354d0b84b066a5813b79fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:21.125877 7564 tablet_replica.cc:330] T 93e9d5dfce844201a5c673326c88d50e P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: stopping tablet replica
I20260928 06:35:21.125911 7562 raft_consensus.cc:2258] T 93e9d5dfce844201a5c673326c88d50e P 796dc2765a354d0b84b066a5813b79fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:21.126015 7564 raft_consensus.cc:2229] T 93e9d5dfce844201a5c673326c88d50e P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:21.126149 7564 raft_consensus.cc:2258] T 93e9d5dfce844201a5c673326c88d50e P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:21.126719 7562 ts_tablet_manager.cc:1905] T 93e9d5dfce844201a5c673326c88d50e P 796dc2765a354d0b84b066a5813b79fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:21.126796 7564 ts_tablet_manager.cc:1905] T 93e9d5dfce844201a5c673326c88d50e P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:21.129410 7564 ts_tablet_manager.cc:1918] T 93e9d5dfce844201a5c673326c88d50e P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:35:21.129410 7562 ts_tablet_manager.cc:1918] T 93e9d5dfce844201a5c673326c88d50e P 796dc2765a354d0b84b066a5813b79fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:35:21.129491 7562 log.cc:1196] T 93e9d5dfce844201a5c673326c88d50e P 796dc2765a354d0b84b066a5813b79fc: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-1/wal/wals/93e9d5dfce844201a5c673326c88d50e
I20260928 06:35:21.129491 7564 log.cc:1196] T 93e9d5dfce844201a5c673326c88d50e P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/93e9d5dfce844201a5c673326c88d50e
I20260928 06:35:21.129771 7562 ts_tablet_manager.cc:1939] T 93e9d5dfce844201a5c673326c88d50e P 796dc2765a354d0b84b066a5813b79fc: Deleting consensus metadata
I20260928 06:35:21.129772 7564 ts_tablet_manager.cc:1939] T 93e9d5dfce844201a5c673326c88d50e P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Deleting consensus metadata
I20260928 06:35:21.130710 6946 catalog_manager.cc:4841] TS 796dc2765a354d0b84b066a5813b79fc (127.5.22.2:35601): tablet 93e9d5dfce844201a5c673326c88d50e (table test-table [id=2a2f7ba1631f4bcba607c48dce7fcf1f]) successfully deleted
I20260928 06:35:21.130705 6947 catalog_manager.cc:4841] TS 7ecd8bc05f2747b8ae7e7a3bbe2f8dad (127.5.22.3:40259): tablet 93e9d5dfce844201a5c673326c88d50e (table test-table [id=2a2f7ba1631f4bcba607c48dce7fcf1f]) successfully deleted
I20260928 06:35:21.134646 7563 tablet_replica.cc:330] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1: stopping tablet replica
I20260928 06:35:21.134814 7563 raft_consensus.cc:2229] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:35:21.135231 7563 raft_consensus.cc:2258] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:21.136005 7563 ts_tablet_manager.cc:1905] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:21.139338 7563 ts_tablet_manager.cc:1918] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:35:21.139424 7563 log.cc:1196] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-0/wal/wals/93e9d5dfce844201a5c673326c88d50e
I20260928 06:35:21.139752 7563 ts_tablet_manager.cc:1939] T 93e9d5dfce844201a5c673326c88d50e P 4f742b414fa349beada5febca578eda1: Deleting consensus metadata
I20260928 06:35:21.140647 6948 catalog_manager.cc:4841] TS 4f742b414fa349beada5febca578eda1 (127.5.22.1:37213): tablet 93e9d5dfce844201a5c673326c88d50e (table test-table [id=2a2f7ba1631f4bcba607c48dce7fcf1f]) successfully deleted
Sep 28 06:35:21 dist-test-slave-j0qf krb5kdc[6916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577321, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:35:21 dist-test-slave-j0qf krb5kdc[6916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577321, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:21.160459 7572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:21.147697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59682 (local address 127.5.22.62:34005)
0928 06:35:21.148096 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:21.148103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:21.148118 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:35:21.148180 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:21.148183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:21.148223 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:21.148298 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:21.148938 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:21.149499 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:21.150469 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:21.150711 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:21.150805 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:21.156951 (+ 6146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:21.156955 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:21.156957 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:21.156984 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:21.158473 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:21.158893 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:21.158895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:21.158896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:21.158972 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:21.159292 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:21.159296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:21.159298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:21.159553 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:21.159710 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:21.160114 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 06:35:21.160249 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":180,"threads_started":1}
I20260928 06:35:21.163461 6961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:35:21.163874 6961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:21.168725 7100 tablet_service.cc:1467] Processing CreateTablet for tablet 273a3ef6059a4bf78c83c6a81942c9e3 (DEFAULT_TABLE table=test-table [id=c3d9a947bde4457aafb2ec50ad840029]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:21.168738 7241 tablet_service.cc:1467] Processing CreateTablet for tablet 273a3ef6059a4bf78c83c6a81942c9e3 (DEFAULT_TABLE table=test-table [id=c3d9a947bde4457aafb2ec50ad840029]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:21.168923 7387 tablet_service.cc:1467] Processing CreateTablet for tablet 273a3ef6059a4bf78c83c6a81942c9e3 (DEFAULT_TABLE table=test-table [id=c3d9a947bde4457aafb2ec50ad840029]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:21.169041 7241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 273a3ef6059a4bf78c83c6a81942c9e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:21.169041 7100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 273a3ef6059a4bf78c83c6a81942c9e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:21.169224 7387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 273a3ef6059a4bf78c83c6a81942c9e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:21.171391 7573 tablet_bootstrap.cc:492] T 273a3ef6059a4bf78c83c6a81942c9e3 P 4f742b414fa349beada5febca578eda1: Bootstrap starting.
I20260928 06:35:21.171761 7575 tablet_bootstrap.cc:492] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Bootstrap starting.
I20260928 06:35:21.172120 7576 tablet_bootstrap.cc:492] T 273a3ef6059a4bf78c83c6a81942c9e3 P 796dc2765a354d0b84b066a5813b79fc: Bootstrap starting.
I20260928 06:35:21.172432 7573 tablet_bootstrap.cc:654] T 273a3ef6059a4bf78c83c6a81942c9e3 P 4f742b414fa349beada5febca578eda1: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:21.173075 7575 tablet_bootstrap.cc:654] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:21.173384 7576 tablet_bootstrap.cc:654] T 273a3ef6059a4bf78c83c6a81942c9e3 P 796dc2765a354d0b84b066a5813b79fc: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:21.173950 7573 tablet_bootstrap.cc:492] T 273a3ef6059a4bf78c83c6a81942c9e3 P 4f742b414fa349beada5febca578eda1: No bootstrap required, opened a new log
I20260928 06:35:21.174067 7573 ts_tablet_manager.cc:1397] T 273a3ef6059a4bf78c83c6a81942c9e3 P 4f742b414fa349beada5febca578eda1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:35:21.174391 7575 tablet_bootstrap.cc:492] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: No bootstrap required, opened a new log
I20260928 06:35:21.174482 7575 ts_tablet_manager.cc:1397] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:35:21.174525 7573 raft_consensus.cc:348] T 273a3ef6059a4bf78c83c6a81942c9e3 P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:21.174641 7573 raft_consensus.cc:374] T 273a3ef6059a4bf78c83c6a81942c9e3 P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:21.174671 7573 raft_consensus.cc:729] T 273a3ef6059a4bf78c83c6a81942c9e3 P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f742b414fa349beada5febca578eda1, State: Initialized, Role: FOLLOWER
I20260928 06:35:21.174796 7573 consensus_queue.cc:260] T 273a3ef6059a4bf78c83c6a81942c9e3 P 4f742b414fa349beada5febca578eda1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:21.174945 7575 raft_consensus.cc:348] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:21.175051 7575 raft_consensus.cc:374] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:21.175084 7575 raft_consensus.cc:729] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ecd8bc05f2747b8ae7e7a3bbe2f8dad, State: Initialized, Role: FOLLOWER
I20260928 06:35:21.175186 7576 tablet_bootstrap.cc:492] T 273a3ef6059a4bf78c83c6a81942c9e3 P 796dc2765a354d0b84b066a5813b79fc: No bootstrap required, opened a new log
I20260928 06:35:21.175220 7573 ts_tablet_manager.cc:1428] T 273a3ef6059a4bf78c83c6a81942c9e3 P 4f742b414fa349beada5febca578eda1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:21.175263 7576 ts_tablet_manager.cc:1397] T 273a3ef6059a4bf78c83c6a81942c9e3 P 796dc2765a354d0b84b066a5813b79fc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:35:21.175225 7575 consensus_queue.cc:260] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:21.175621 7575 ts_tablet_manager.cc:1428] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:21.175751 7576 raft_consensus.cc:348] T 273a3ef6059a4bf78c83c6a81942c9e3 P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:21.175872 7576 raft_consensus.cc:374] T 273a3ef6059a4bf78c83c6a81942c9e3 P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:21.175913 7576 raft_consensus.cc:729] T 273a3ef6059a4bf78c83c6a81942c9e3 P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 796dc2765a354d0b84b066a5813b79fc, State: Initialized, Role: FOLLOWER
I20260928 06:35:21.176050 7576 consensus_queue.cc:260] T 273a3ef6059a4bf78c83c6a81942c9e3 P 796dc2765a354d0b84b066a5813b79fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:21.176342 7576 ts_tablet_manager.cc:1428] T 273a3ef6059a4bf78c83c6a81942c9e3 P 796dc2765a354d0b84b066a5813b79fc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:35:21.191690 7580 raft_consensus.cc:482] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:21.191875 7580 raft_consensus.cc:504] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:21.193130 7580 leader_election.cc:288] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f742b414fa349beada5febca578eda1 (127.5.22.1:37213), 796dc2765a354d0b84b066a5813b79fc (127.5.22.2:35601)
I20260928 06:35:21.197953 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:21.193479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:35147 (local address 127.5.22.2:35601)
0928 06:35:21.193769 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:21.193778 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:21.193901 (+ 123us) server_negotiation.cc:407] Connection header received
0928 06:35:21.194050 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:21.194053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:21.194101 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:21.194191 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:21.194526 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:21.195188 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:21.196990 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:21.197387 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:21.197506 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:21.197584 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:21.197753 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 06:35:21.197822 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":97,"threads_started":1}
I20260928 06:35:21.198083 7583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:21.193438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:40415 (local address 127.5.22.1:37213)
0928 06:35:21.193712 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:21.193786 (+ 74us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:21.193805 (+ 19us) server_negotiation.cc:407] Connection header received
0928 06:35:21.193965 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:21.193968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:21.194021 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:21.194117 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:21.194457 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:21.195192 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:21.197045 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:21.197481 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:21.197617 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:21.197681 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:21.197870 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 06:35:21.197946 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":108,"threads_started":1}
I20260928 06:35:21.197966 7581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:21.193383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:35601 (local address 127.5.22.3:35147)
0928 06:35:21.193804 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:21.193817 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:21.193943 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:21.194209 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:21.194212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:21.194227 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:21.194413 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:21.194419 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:21.195425 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:21.195427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:21.196874 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:21.196885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:21.197508 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:21.197512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:21.197615 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:21.197638 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 06:35:21.197728 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 06:35:21.197803 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":28,"thread_start_us":147,"threads_started":1}
I20260928 06:35:21.198117 7582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:21.193308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:37213 (local address 127.5.22.3:40415)
0928 06:35:21.193682 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:21.193698 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:21.193832 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:21.194129 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:21.194132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:21.194148 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:21.194294 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:21.194300 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:21.195332 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:21.195336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:21.196878 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:21.196885 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:21.197633 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:21.197637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:21.197735 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:21.197756 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 06:35:21.197854 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 06:35:21.197927 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":45,"thread_start_us":107,"threads_started":1}
I20260928 06:35:21.198797 7262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "273a3ef6059a4bf78c83c6a81942c9e3" candidate_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "796dc2765a354d0b84b066a5813b79fc" is_pre_election: true
I20260928 06:35:21.198799 7119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "273a3ef6059a4bf78c83c6a81942c9e3" candidate_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f742b414fa349beada5febca578eda1" is_pre_election: true
I20260928 06:35:21.198961 7119 raft_consensus.cc:2454] T 273a3ef6059a4bf78c83c6a81942c9e3 P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecd8bc05f2747b8ae7e7a3bbe2f8dad in term 0.
I20260928 06:35:21.198975 7262 raft_consensus.cc:2454] T 273a3ef6059a4bf78c83c6a81942c9e3 P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecd8bc05f2747b8ae7e7a3bbe2f8dad in term 0.
I20260928 06:35:21.199342 7339 leader_election.cc:302] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f742b414fa349beada5febca578eda1, 7ecd8bc05f2747b8ae7e7a3bbe2f8dad; no voters:
I20260928 06:35:21.199635 7580 raft_consensus.cc:2781] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:21.199700 7580 raft_consensus.cc:482] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:21.199728 7580 raft_consensus.cc:3037] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:21.200999 7580 raft_consensus.cc:504] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:21.201398 7580 leader_election.cc:288] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [CANDIDATE]: Term 1 election: Requested vote from peers 4f742b414fa349beada5febca578eda1 (127.5.22.1:37213), 796dc2765a354d0b84b066a5813b79fc (127.5.22.2:35601)
I20260928 06:35:21.201768 7119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "273a3ef6059a4bf78c83c6a81942c9e3" candidate_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f742b414fa349beada5febca578eda1"
I20260928 06:35:21.201908 7119 raft_consensus.cc:3037] T 273a3ef6059a4bf78c83c6a81942c9e3 P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:21.201949 7262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "273a3ef6059a4bf78c83c6a81942c9e3" candidate_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "796dc2765a354d0b84b066a5813b79fc"
I20260928 06:35:21.202090 7262 raft_consensus.cc:3037] T 273a3ef6059a4bf78c83c6a81942c9e3 P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:21.203118 7119 raft_consensus.cc:2454] T 273a3ef6059a4bf78c83c6a81942c9e3 P 4f742b414fa349beada5febca578eda1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecd8bc05f2747b8ae7e7a3bbe2f8dad in term 1.
I20260928 06:35:21.203136 7262 raft_consensus.cc:2454] T 273a3ef6059a4bf78c83c6a81942c9e3 P 796dc2765a354d0b84b066a5813b79fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecd8bc05f2747b8ae7e7a3bbe2f8dad in term 1.
I20260928 06:35:21.203393 7339 leader_election.cc:302] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f742b414fa349beada5febca578eda1, 7ecd8bc05f2747b8ae7e7a3bbe2f8dad; no voters:
I20260928 06:35:21.203636 7580 raft_consensus.cc:2781] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:21.203822 7580 raft_consensus.cc:686] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 1 LEADER]: Becoming Leader. State: Replica: 7ecd8bc05f2747b8ae7e7a3bbe2f8dad, State: Running, Role: LEADER
I20260928 06:35:21.204035 7580 consensus_queue.cc:237] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:21.206542 6962 catalog_manager.cc:5495] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ecd8bc05f2747b8ae7e7a3bbe2f8dad (127.5.22.3). New cstate: current_term: 1 leader_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } health_report { overall_health: HEALTHY } } }
I20260928 06:35:21.225464 7588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:21.221281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46864 (local address 127.5.22.3:40259)
0928 06:35:21.221639 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:21.221666 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:21.221680 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:35:21.221750 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:21.221751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:21.221802 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:21.221934 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:21.222200 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:21.223057 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:21.223886 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:21.224219 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:21.224305 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:21.224548 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:21.224998 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:21.225090 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:21.225155 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 06:35:21.225204 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":111,"threads_started":1}
I20260928 06:35:21.236402 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:21.232818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:39307 (local address 127.5.22.2:35601)
0928 06:35:21.232975 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:21.232978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:21.233009 (+ 31us) server_negotiation.cc:407] Connection header received
0928 06:35:21.233135 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:21.233137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:21.233181 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:21.233248 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:21.233624 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:21.234124 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:21.235295 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:21.235507 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:21.235663 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:21.236008 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:21.236088 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:21.236148 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:21.236256 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 06:35:21.236300 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 06:35:21.236446 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:21.232616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:35601 (local address 127.5.22.3:39307)
0928 06:35:21.232950 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:21.232963 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:21.233044 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:21.233323 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:21.233325 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:21.233333 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:35:21.233449 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:21.233454 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:21.234267 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:21.234270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:21.235176 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:21.235186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:21.235656 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:21.235659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:21.235774 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:21.235818 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:35:21.236167 (+ 349us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:35:21.236172 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:35:21.236223 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:35:21.236287 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":132,"threads_started":1}
I20260928 06:35:21.242970 7262 raft_consensus.cc:1261] T 273a3ef6059a4bf78c83c6a81942c9e3 P 796dc2765a354d0b84b066a5813b79fc [term 1 FOLLOWER]: Refusing update from remote peer 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:21.243417 7119 raft_consensus.cc:1261] T 273a3ef6059a4bf78c83c6a81942c9e3 P 4f742b414fa349beada5febca578eda1 [term 1 FOLLOWER]: Refusing update from remote peer 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:21.243608 7585 consensus_queue.cc:1035] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [LEADER]: Connected to new peer: Peer: permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:21.243893 7580 consensus_queue.cc:1035] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:21.262261 7588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:21.258730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:53127 (local address 127.5.22.3:40259)
0928 06:35:21.258877 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:21.258882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:21.259054 (+ 172us) server_negotiation.cc:407] Connection header received
0928 06:35:21.259204 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:21.259207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:21.259251 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:21.259334 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:21.259586 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:21.260101 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:21.260972 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:21.261274 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:21.261420 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:21.261755 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:21.261868 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:21.261922 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:21.262084 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 06:35:21.262144 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 06:35:21.262353 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:21.258618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:40259 (local address 127.5.22.2:53127)
0928 06:35:21.258962 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:21.258975 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:21.259103 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:21.259354 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:21.259358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:21.259366 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:35:21.259468 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:21.259474 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:21.260210 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:21.260216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:21.260867 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:21.260879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:21.261418 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:21.261422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:21.261516 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:21.261552 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:35:21.261994 (+ 442us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:35:21.261999 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:35:21.262062 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 06:35:21.262156 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":114,"threads_started":1}
I20260928 06:35:21.285790 6960 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:21.285980 6960 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:21.288259 6960 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4: Sending DeleteTablet for 3 replicas of tablet 273a3ef6059a4bf78c83c6a81942c9e3
I20260928 06:35:21.288865 7100 tablet_service.cc:1514] Processing DeleteTablet for tablet 273a3ef6059a4bf78c83c6a81942c9e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:21 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:54856
I20260928 06:35:21.288918 7241 tablet_service.cc:1514] Processing DeleteTablet for tablet 273a3ef6059a4bf78c83c6a81942c9e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:21 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:44002
I20260928 06:35:21.289098 7387 tablet_service.cc:1514] Processing DeleteTablet for tablet 273a3ef6059a4bf78c83c6a81942c9e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:21 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:46848
I20260928 06:35:21.289161 7562 tablet_replica.cc:330] T 273a3ef6059a4bf78c83c6a81942c9e3 P 796dc2765a354d0b84b066a5813b79fc: stopping tablet replica
I20260928 06:35:21.289294 7563 tablet_replica.cc:330] T 273a3ef6059a4bf78c83c6a81942c9e3 P 4f742b414fa349beada5febca578eda1: stopping tablet replica
I20260928 06:35:21.289290 7562 raft_consensus.cc:2229] T 273a3ef6059a4bf78c83c6a81942c9e3 P 796dc2765a354d0b84b066a5813b79fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:21.289415 7562 raft_consensus.cc:2258] T 273a3ef6059a4bf78c83c6a81942c9e3 P 796dc2765a354d0b84b066a5813b79fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:21.289417 7563 raft_consensus.cc:2229] T 273a3ef6059a4bf78c83c6a81942c9e3 P 4f742b414fa349beada5febca578eda1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:21.289529 7563 raft_consensus.cc:2258] T 273a3ef6059a4bf78c83c6a81942c9e3 P 4f742b414fa349beada5febca578eda1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:21.289717 7564 tablet_replica.cc:330] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: stopping tablet replica
I20260928 06:35:21.289853 7564 raft_consensus.cc:2229] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:35:21.290290 7562 ts_tablet_manager.cc:1905] T 273a3ef6059a4bf78c83c6a81942c9e3 P 796dc2765a354d0b84b066a5813b79fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:21.290279 7564 raft_consensus.cc:2258] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:21.291023 7564 ts_tablet_manager.cc:1905] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:21.291111 7563 ts_tablet_manager.cc:1905] T 273a3ef6059a4bf78c83c6a81942c9e3 P 4f742b414fa349beada5febca578eda1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:21.293319 7562 ts_tablet_manager.cc:1918] T 273a3ef6059a4bf78c83c6a81942c9e3 P 796dc2765a354d0b84b066a5813b79fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:35:21.293388 7562 log.cc:1196] T 273a3ef6059a4bf78c83c6a81942c9e3 P 796dc2765a354d0b84b066a5813b79fc: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-1/wal/wals/273a3ef6059a4bf78c83c6a81942c9e3
I20260928 06:35:21.293721 7562 ts_tablet_manager.cc:1939] T 273a3ef6059a4bf78c83c6a81942c9e3 P 796dc2765a354d0b84b066a5813b79fc: Deleting consensus metadata
I20260928 06:35:21.294009 7564 ts_tablet_manager.cc:1918] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:35:21.294061 7563 ts_tablet_manager.cc:1918] T 273a3ef6059a4bf78c83c6a81942c9e3 P 4f742b414fa349beada5febca578eda1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:35:21.294092 7564 log.cc:1196] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/273a3ef6059a4bf78c83c6a81942c9e3
I20260928 06:35:21.294133 7563 log.cc:1196] T 273a3ef6059a4bf78c83c6a81942c9e3 P 4f742b414fa349beada5febca578eda1: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-0/wal/wals/273a3ef6059a4bf78c83c6a81942c9e3
I20260928 06:35:21.294454 7563 ts_tablet_manager.cc:1939] T 273a3ef6059a4bf78c83c6a81942c9e3 P 4f742b414fa349beada5febca578eda1: Deleting consensus metadata
I20260928 06:35:21.294570 6946 catalog_manager.cc:4841] TS 796dc2765a354d0b84b066a5813b79fc (127.5.22.2:35601): tablet 273a3ef6059a4bf78c83c6a81942c9e3 (table test-table [id=c3d9a947bde4457aafb2ec50ad840029]) successfully deleted
I20260928 06:35:21.294701 7564 ts_tablet_manager.cc:1939] T 273a3ef6059a4bf78c83c6a81942c9e3 P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Deleting consensus metadata
I20260928 06:35:21.295222 6948 catalog_manager.cc:4841] TS 4f742b414fa349beada5febca578eda1 (127.5.22.1:37213): tablet 273a3ef6059a4bf78c83c6a81942c9e3 (table test-table [id=c3d9a947bde4457aafb2ec50ad840029]) successfully deleted
I20260928 06:35:21.295463 6947 catalog_manager.cc:4841] TS 7ecd8bc05f2747b8ae7e7a3bbe2f8dad (127.5.22.3:40259): tablet 273a3ef6059a4bf78c83c6a81942c9e3 (table test-table [id=c3d9a947bde4457aafb2ec50ad840029]) successfully deleted
I20260928 06:35:21.296381 6960 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:35:21.296846 6960 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:21.302035 7100 tablet_service.cc:1467] Processing CreateTablet for tablet 8d477fd050344948a4f5508796c2178f (DEFAULT_TABLE table=test-table [id=ef59ddf13a1f464aa1af40e6a6460b1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:21.302142 7241 tablet_service.cc:1467] Processing CreateTablet for tablet 8d477fd050344948a4f5508796c2178f (DEFAULT_TABLE table=test-table [id=ef59ddf13a1f464aa1af40e6a6460b1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:21.302304 7100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d477fd050344948a4f5508796c2178f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:21.302255 7387 tablet_service.cc:1467] Processing CreateTablet for tablet 8d477fd050344948a4f5508796c2178f (DEFAULT_TABLE table=test-table [id=ef59ddf13a1f464aa1af40e6a6460b1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:21.302438 7241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d477fd050344948a4f5508796c2178f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:21.302549 7387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d477fd050344948a4f5508796c2178f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:21.304636 7573 tablet_bootstrap.cc:492] T 8d477fd050344948a4f5508796c2178f P 4f742b414fa349beada5febca578eda1: Bootstrap starting.
I20260928 06:35:21.305014 7575 tablet_bootstrap.cc:492] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Bootstrap starting.
I20260928 06:35:21.305734 7573 tablet_bootstrap.cc:654] T 8d477fd050344948a4f5508796c2178f P 4f742b414fa349beada5febca578eda1: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:21.306445 7575 tablet_bootstrap.cc:654] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:21.307430 7576 tablet_bootstrap.cc:492] T 8d477fd050344948a4f5508796c2178f P 796dc2765a354d0b84b066a5813b79fc: Bootstrap starting.
I20260928 06:35:21.308312 7576 tablet_bootstrap.cc:654] T 8d477fd050344948a4f5508796c2178f P 796dc2765a354d0b84b066a5813b79fc: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:21.308406 7573 tablet_bootstrap.cc:492] T 8d477fd050344948a4f5508796c2178f P 4f742b414fa349beada5febca578eda1: No bootstrap required, opened a new log
I20260928 06:35:21.308509 7573 ts_tablet_manager.cc:1397] T 8d477fd050344948a4f5508796c2178f P 4f742b414fa349beada5febca578eda1: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260928 06:35:21.309003 7573 raft_consensus.cc:348] T 8d477fd050344948a4f5508796c2178f P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:21.309190 7573 raft_consensus.cc:374] T 8d477fd050344948a4f5508796c2178f P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:21.309242 7573 raft_consensus.cc:729] T 8d477fd050344948a4f5508796c2178f P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f742b414fa349beada5febca578eda1, State: Initialized, Role: FOLLOWER
I20260928 06:35:21.309372 7573 consensus_queue.cc:260] T 8d477fd050344948a4f5508796c2178f P 4f742b414fa349beada5febca578eda1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:21.309680 7573 ts_tablet_manager.cc:1428] T 8d477fd050344948a4f5508796c2178f P 4f742b414fa349beada5febca578eda1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:21.310391 7576 tablet_bootstrap.cc:492] T 8d477fd050344948a4f5508796c2178f P 796dc2765a354d0b84b066a5813b79fc: No bootstrap required, opened a new log
I20260928 06:35:21.310488 7576 ts_tablet_manager.cc:1397] T 8d477fd050344948a4f5508796c2178f P 796dc2765a354d0b84b066a5813b79fc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:35:21.310930 7575 tablet_bootstrap.cc:492] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: No bootstrap required, opened a new log
I20260928 06:35:21.310853 7576 raft_consensus.cc:348] T 8d477fd050344948a4f5508796c2178f P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:21.311056 7575 ts_tablet_manager.cc:1397] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260928 06:35:21.311093 7576 raft_consensus.cc:374] T 8d477fd050344948a4f5508796c2178f P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:21.311146 7576 raft_consensus.cc:729] T 8d477fd050344948a4f5508796c2178f P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 796dc2765a354d0b84b066a5813b79fc, State: Initialized, Role: FOLLOWER
I20260928 06:35:21.311275 7576 consensus_queue.cc:260] T 8d477fd050344948a4f5508796c2178f P 796dc2765a354d0b84b066a5813b79fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:21.311543 7576 ts_tablet_manager.cc:1428] T 8d477fd050344948a4f5508796c2178f P 796dc2765a354d0b84b066a5813b79fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:21.318213 7575 raft_consensus.cc:348] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:21.318401 7575 raft_consensus.cc:374] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:21.318454 7575 raft_consensus.cc:729] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ecd8bc05f2747b8ae7e7a3bbe2f8dad, State: Initialized, Role: FOLLOWER
I20260928 06:35:21.318602 7575 consensus_queue.cc:260] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:21.318924 7575 ts_tablet_manager.cc:1428] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20260928 06:35:21.432390 7580 raft_consensus.cc:482] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:21.432555 7580 raft_consensus.cc:504] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:21.433008 7580 leader_election.cc:288] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f742b414fa349beada5febca578eda1 (127.5.22.1:37213), 796dc2765a354d0b84b066a5813b79fc (127.5.22.2:35601)
I20260928 06:35:21.433467 7262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d477fd050344948a4f5508796c2178f" candidate_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "796dc2765a354d0b84b066a5813b79fc" is_pre_election: true
I20260928 06:35:21.433468 7119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d477fd050344948a4f5508796c2178f" candidate_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f742b414fa349beada5febca578eda1" is_pre_election: true
I20260928 06:35:21.433645 7262 raft_consensus.cc:2454] T 8d477fd050344948a4f5508796c2178f P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecd8bc05f2747b8ae7e7a3bbe2f8dad in term 0.
I20260928 06:35:21.433645 7119 raft_consensus.cc:2454] T 8d477fd050344948a4f5508796c2178f P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecd8bc05f2747b8ae7e7a3bbe2f8dad in term 0.
I20260928 06:35:21.434042 7337 leader_election.cc:302] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 796dc2765a354d0b84b066a5813b79fc, 7ecd8bc05f2747b8ae7e7a3bbe2f8dad; no voters:
I20260928 06:35:21.434295 7580 raft_consensus.cc:2781] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:21.434453 7580 raft_consensus.cc:482] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:21.434487 7580 raft_consensus.cc:3037] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:21.435663 7580 raft_consensus.cc:504] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:21.436062 7580 leader_election.cc:288] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [CANDIDATE]: Term 1 election: Requested vote from peers 4f742b414fa349beada5febca578eda1 (127.5.22.1:37213), 796dc2765a354d0b84b066a5813b79fc (127.5.22.2:35601)
I20260928 06:35:21.436468 7262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d477fd050344948a4f5508796c2178f" candidate_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "796dc2765a354d0b84b066a5813b79fc"
I20260928 06:35:21.436470 7119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d477fd050344948a4f5508796c2178f" candidate_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f742b414fa349beada5febca578eda1"
I20260928 06:35:21.436604 7262 raft_consensus.cc:3037] T 8d477fd050344948a4f5508796c2178f P 796dc2765a354d0b84b066a5813b79fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:21.436604 7119 raft_consensus.cc:3037] T 8d477fd050344948a4f5508796c2178f P 4f742b414fa349beada5febca578eda1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:21.437786 7119 raft_consensus.cc:2454] T 8d477fd050344948a4f5508796c2178f P 4f742b414fa349beada5febca578eda1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecd8bc05f2747b8ae7e7a3bbe2f8dad in term 1.
I20260928 06:35:21.437786 7262 raft_consensus.cc:2454] T 8d477fd050344948a4f5508796c2178f P 796dc2765a354d0b84b066a5813b79fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecd8bc05f2747b8ae7e7a3bbe2f8dad in term 1.
I20260928 06:35:21.438118 7337 leader_election.cc:302] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 796dc2765a354d0b84b066a5813b79fc, 7ecd8bc05f2747b8ae7e7a3bbe2f8dad; no voters:
I20260928 06:35:21.438434 7580 raft_consensus.cc:2781] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:21.438788 7580 raft_consensus.cc:686] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 1 LEADER]: Becoming Leader. State: Replica: 7ecd8bc05f2747b8ae7e7a3bbe2f8dad, State: Running, Role: LEADER
I20260928 06:35:21.438938 7580 consensus_queue.cc:237] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } }
I20260928 06:35:21.441036 6961 catalog_manager.cc:5495] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ecd8bc05f2747b8ae7e7a3bbe2f8dad (127.5.22.3). New cstate: current_term: 1 leader_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecd8bc05f2747b8ae7e7a3bbe2f8dad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40259 } health_report { overall_health: HEALTHY } } }
I20260928 06:35:21.474727 7262 raft_consensus.cc:1261] T 8d477fd050344948a4f5508796c2178f P 796dc2765a354d0b84b066a5813b79fc [term 1 FOLLOWER]: Refusing update from remote peer 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:21.474833 7119 raft_consensus.cc:1261] T 8d477fd050344948a4f5508796c2178f P 4f742b414fa349beada5febca578eda1 [term 1 FOLLOWER]: Refusing update from remote peer 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:21.475184 7580 consensus_queue.cc:1035] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [LEADER]: Connected to new peer: Peer: permanent_uuid: "796dc2765a354d0b84b066a5813b79fc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:21.475325 7594 consensus_queue.cc:1035] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f742b414fa349beada5febca578eda1" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 37213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 06:35:21.489818 7544 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 06:35:23.039484 6960 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:23.039640 6960 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:23.042069 6960 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf19afcf2f4149989eb8369a994624f4: Sending DeleteTablet for 3 replicas of tablet 8d477fd050344948a4f5508796c2178f
I20260928 06:35:23.042728 7241 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d477fd050344948a4f5508796c2178f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:23 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:44002
I20260928 06:35:23.042762 7387 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d477fd050344948a4f5508796c2178f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:23 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:46848
I20260928 06:35:23.042914 7100 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d477fd050344948a4f5508796c2178f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:23 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:54856
I20260928 06:35:23.043131 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 7034
I20260928 06:35:23.044777 7626 tablet_replica.cc:330] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: stopping tablet replica
I20260928 06:35:23.044926 7626 raft_consensus.cc:2229] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:35:23.045147 7626 raft_consensus.cc:2258] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:23.045318 7625 tablet_replica.cc:330] T 8d477fd050344948a4f5508796c2178f P 796dc2765a354d0b84b066a5813b79fc: stopping tablet replica
I20260928 06:35:23.045467 7625 raft_consensus.cc:2229] T 8d477fd050344948a4f5508796c2178f P 796dc2765a354d0b84b066a5813b79fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:23.045588 7625 raft_consensus.cc:2258] T 8d477fd050344948a4f5508796c2178f P 796dc2765a354d0b84b066a5813b79fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:23.045899 7626 ts_tablet_manager.cc:1905] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:23.046195 7625 ts_tablet_manager.cc:1905] T 8d477fd050344948a4f5508796c2178f P 796dc2765a354d0b84b066a5813b79fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:23.048250 7625 ts_tablet_manager.cc:1918] T 8d477fd050344948a4f5508796c2178f P 796dc2765a354d0b84b066a5813b79fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:35:23.048349 7625 log.cc:1196] T 8d477fd050344948a4f5508796c2178f P 796dc2765a354d0b84b066a5813b79fc: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-1/wal/wals/8d477fd050344948a4f5508796c2178f
I20260928 06:35:23.048609 7625 ts_tablet_manager.cc:1939] T 8d477fd050344948a4f5508796c2178f P 796dc2765a354d0b84b066a5813b79fc: Deleting consensus metadata
I20260928 06:35:23.049304 7626 ts_tablet_manager.cc:1918] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:35:23.049291 6946 catalog_manager.cc:4841] TS 796dc2765a354d0b84b066a5813b79fc (127.5.22.2:35601): tablet 8d477fd050344948a4f5508796c2178f (table test-table [id=ef59ddf13a1f464aa1af40e6a6460b1d]) successfully deleted
I20260928 06:35:23.049386 7626 log.cc:1196] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/8d477fd050344948a4f5508796c2178f
I20260928 06:35:23.049686 7626 ts_tablet_manager.cc:1939] T 8d477fd050344948a4f5508796c2178f P 7ecd8bc05f2747b8ae7e7a3bbe2f8dad: Deleting consensus metadata
I20260928 06:35:23.050451 6947 catalog_manager.cc:4841] TS 7ecd8bc05f2747b8ae7e7a3bbe2f8dad (127.5.22.3:40259): tablet 8d477fd050344948a4f5508796c2178f (table test-table [id=ef59ddf13a1f464aa1af40e6a6460b1d]) successfully deleted
I20260928 06:35:23.051795 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 7175
W20260928 06:35:23.051795 6948 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.22.1:37213 (error 108)
I20260928 06:35:23.059445 7629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:23.058999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:37213 (local address 127.0.0.1:54880)
0928 06:35:23.059280 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:23.059355 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.1:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":113,"threads_started":1}
W20260928 06:35:23.059813 6948 catalog_manager.cc:4584] TS 4f742b414fa349beada5febca578eda1 (127.5.22.1:37213): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d477fd050344948a4f5508796c2178f: Network error: Client connection negotiation failed: client connection to 127.5.22.1:37213: connect: Connection refused (error 111)
I20260928 06:35:23.060271 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 7320
I20260928 06:35:23.067504 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 6929
2026-09-28T06:35:23Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6576 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:23 dist-test-slave-j0qf krb5kdc[7632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:23 dist-test-slave-j0qf krb5kdc[7632](info): set up 2 sockets
Sep 28 06:35:23 dist-test-slave-j0qf krb5kdc[7632](info): commencing operation
krb5kdc: starting...
W20260928 06:35:24.117074 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:35:24 dist-test-slave-j0qf krb5kdc[7632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:35:24.242445 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:38427
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:38427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:24.352233 7645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:24.352555 7645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:24.352630 7645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:24.356606 7645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:24.356694 7645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:24.356732 7645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:24.356789 7645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:24.356827 7645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:24.362221 7645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:38427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:38427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:24.363320 7645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:24.364375 7645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:24.373378 7651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:24.373339 7653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:24.373353 7650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:24.373521 7645 server_base.cc:1034] running on GCE node
I20260928 06:35:24.374423 7645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:24.375726 7645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:24.376915 7645 hybrid_clock.cc:648] HybridClock initialized: now 1790577324376865 us; error 60 us; skew 500 ppm
Sep 28 06:35:24 dist-test-slave-j0qf krb5kdc[7632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577324, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:24.380075 7645 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:35:24.381000 7645 webserver.cc:460] Webserver started at http://127.5.22.62:41841/ using document root <none> and password file <none>
I20260928 06:35:24.381608 7645 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:24.381673 7645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:24.381924 7645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:24.384101 7645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "d5172f92b5dc4d49b9c018d78f41035a"
format_stamp: "Formatted at 2026-09-28 06:35:24 on dist-test-slave-j0qf"
server_key: "00b27adeefa4f4135ff582e469ea34df"
server_key_iv: "f6e2e658301e858037bc3738f41a8b6a"
server_key_version: "encryptionkey@0"
I20260928 06:35:24.384680 7645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "d5172f92b5dc4d49b9c018d78f41035a"
format_stamp: "Formatted at 2026-09-28 06:35:24 on dist-test-slave-j0qf"
server_key: "00b27adeefa4f4135ff582e469ea34df"
server_key_iv: "f6e2e658301e858037bc3738f41a8b6a"
server_key_version: "encryptionkey@0"
I20260928 06:35:24.388550 7645 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260928 06:35:24.391098 7660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:24.392445 7645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:24.392622 7645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "d5172f92b5dc4d49b9c018d78f41035a"
format_stamp: "Formatted at 2026-09-28 06:35:24 on dist-test-slave-j0qf"
server_key: "00b27adeefa4f4135ff582e469ea34df"
server_key_iv: "f6e2e658301e858037bc3738f41a8b6a"
server_key_version: "encryptionkey@0"
I20260928 06:35:24.392747 7645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:24.424158 7645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:24.427740 7645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:24.427996 7645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:24.435966 7645 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:38427
I20260928 06:35:24.435999 7713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:38427 every 8 connection(s)
I20260928 06:35:24.437026 7645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:24.439095 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 7645
I20260928 06:35:24.439198 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:35:24.439551 5208 external_mini_cluster.cc:1428] Setting key 2a9850f4c58ede3975dfa8ce43c01ef5
I20260928 06:35:24.440013 7714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:24.447180 7714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5172f92b5dc4d49b9c018d78f41035a: Bootstrap starting.
Sep 28 06:35:24 dist-test-slave-j0qf krb5kdc[7632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:24.450052 7714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5172f92b5dc4d49b9c018d78f41035a: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:24.451079 7714 log.cc:824] T 00000000000000000000000000000000 P d5172f92b5dc4d49b9c018d78f41035a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:24.453414 7714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5172f92b5dc4d49b9c018d78f41035a: No bootstrap required, opened a new log
I20260928 06:35:24.455065 7717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:24.441014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33772 (local address 127.5.22.62:38427)
0928 06:35:24.441465 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:24.441491 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:24.441530 (+ 39us) server_negotiation.cc:407] Connection header received
0928 06:35:24.442237 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:24.442253 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:24.442673 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:24.442980 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:24.443958 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:24.445094 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:24.446395 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:24.446961 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:24.447087 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:24.449901 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:24.449910 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:24.449930 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:24.449969 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:24.451962 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:24.452378 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:24.452384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:24.452389 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:24.452527 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:24.452805 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:24.452810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:24.452813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:24.453307 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:24.453496 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:24.453825 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 06:35:24.454123 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":168,"threads_started":1}
I20260928 06:35:24.457727 7714 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5172f92b5dc4d49b9c018d78f41035a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5172f92b5dc4d49b9c018d78f41035a" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38427 } }
I20260928 06:35:24.457942 7714 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5172f92b5dc4d49b9c018d78f41035a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:24.457978 7714 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5172f92b5dc4d49b9c018d78f41035a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5172f92b5dc4d49b9c018d78f41035a, State: Initialized, Role: FOLLOWER
I20260928 06:35:24.458387 7714 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5172f92b5dc4d49b9c018d78f41035a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5172f92b5dc4d49b9c018d78f41035a" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38427 } }
I20260928 06:35:24.458524 7714 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5172f92b5dc4d49b9c018d78f41035a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:24.458567 7714 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5172f92b5dc4d49b9c018d78f41035a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:24.458643 7714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5172f92b5dc4d49b9c018d78f41035a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:24.459502 7714 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5172f92b5dc4d49b9c018d78f41035a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5172f92b5dc4d49b9c018d78f41035a" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38427 } }
I20260928 06:35:24.459789 7714 leader_election.cc:302] T 00000000000000000000000000000000 P d5172f92b5dc4d49b9c018d78f41035a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5172f92b5dc4d49b9c018d78f41035a; no voters:
I20260928 06:35:24.460096 7714 leader_election.cc:288] T 00000000000000000000000000000000 P d5172f92b5dc4d49b9c018d78f41035a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:24.460180 7719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5172f92b5dc4d49b9c018d78f41035a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:24.460353 7719 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5172f92b5dc4d49b9c018d78f41035a [term 1 LEADER]: Becoming Leader. State: Replica: d5172f92b5dc4d49b9c018d78f41035a, State: Running, Role: LEADER
I20260928 06:35:24.460654 7719 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5172f92b5dc4d49b9c018d78f41035a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5172f92b5dc4d49b9c018d78f41035a" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38427 } }
I20260928 06:35:24.461380 7714 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5172f92b5dc4d49b9c018d78f41035a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:24.462484 7721 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5172f92b5dc4d49b9c018d78f41035a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5172f92b5dc4d49b9c018d78f41035a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5172f92b5dc4d49b9c018d78f41035a" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38427 } } }
I20260928 06:35:24.462610 7721 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5172f92b5dc4d49b9c018d78f41035a [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:24.462958 7720 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5172f92b5dc4d49b9c018d78f41035a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5172f92b5dc4d49b9c018d78f41035a. Latest consensus state: current_term: 1 leader_uuid: "d5172f92b5dc4d49b9c018d78f41035a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5172f92b5dc4d49b9c018d78f41035a" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38427 } } }
I20260928 06:35:24.463048 7720 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5172f92b5dc4d49b9c018d78f41035a [sys.catalog]: This master's current role is: LEADER
W20260928 06:35:24.466073 7735 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d5172f92b5dc4d49b9c018d78f41035a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:35:24.466148 7735 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:35:24.466238 7728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:24.466998 7728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:24.472808 7728 catalog_manager.cc:1347] Generated new cluster ID: 3d6795cef39e42da8de184dcea229599
I20260928 06:35:24.472934 7728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:24.499266 7728 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:24.500545 7728 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:24.507014 7728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5172f92b5dc4d49b9c018d78f41035a: Generated new TSK 0
I20260928 06:35:24.507866 7728 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.22.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
I20260928 06:35:24.538816 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:38427
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:24.667796 7740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:24.668097 7740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:24.668140 7740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:24.672196 7740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:24.672259 7740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:24.672317 7740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:35:24.677826 7740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:38427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:24.678977 7740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:24.680039 7740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:24.689704 7746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:24.689730 7748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:24.689704 7745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:24.689917 7740 server_base.cc:1034] running on GCE node
I20260928 06:35:24.690451 7740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:24.691035 7740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:24.692190 7740 hybrid_clock.cc:648] HybridClock initialized: now 1790577324692167 us; error 38 us; skew 500 ppm
Sep 28 06:35:24 dist-test-slave-j0qf krb5kdc[7632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577324, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:24.694993 7740 init.cc:377] Logged in from keytab as kudu/127.5.22.1@KRBTEST.COM (short username kudu)
I20260928 06:35:24.695775 7740 webserver.cc:460] Webserver started at http://127.5.22.1:33371/ using document root <none> and password file <none>
I20260928 06:35:24.696277 7740 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:24.696332 7740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:24.696493 7740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:24.698592 7740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "71b7e043cfa94b46b781df0b434501c4"
format_stamp: "Formatted at 2026-09-28 06:35:24 on dist-test-slave-j0qf"
server_key: "cc8446e751eaa063c4ff47823c151a44"
server_key_iv: "d53086691a862f72e06aab062a6a8dca"
server_key_version: "encryptionkey@0"
I20260928 06:35:24.699184 7740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "71b7e043cfa94b46b781df0b434501c4"
format_stamp: "Formatted at 2026-09-28 06:35:24 on dist-test-slave-j0qf"
server_key: "cc8446e751eaa063c4ff47823c151a44"
server_key_iv: "d53086691a862f72e06aab062a6a8dca"
server_key_version: "encryptionkey@0"
I20260928 06:35:24.702901 7740 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:24.705361 7755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:24.706804 7740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:24.706977 7740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "71b7e043cfa94b46b781df0b434501c4"
format_stamp: "Formatted at 2026-09-28 06:35:24 on dist-test-slave-j0qf"
server_key: "cc8446e751eaa063c4ff47823c151a44"
server_key_iv: "d53086691a862f72e06aab062a6a8dca"
server_key_version: "encryptionkey@0"
I20260928 06:35:24.707103 7740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:24.724466 7740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:24.728185 7740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:24.728407 7740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:24.729043 7740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:24.729913 7740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:24.729986 7740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:24.730060 7740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:24.730111 7740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:24.739270 7740 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:33931
I20260928 06:35:24.739348 7868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:33931 every 8 connection(s)
I20260928 06:35:24.740231 7740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:24.747493 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 7740
I20260928 06:35:24.747635 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:24.747912 5208 external_mini_cluster.cc:1428] Setting key e6ae6ccd7bc08a49eed56da8163f306e
Sep 28 06:35:24 dist-test-slave-j0qf krb5kdc[7632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577324, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:24.755307 7717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:24.742281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:44299 (local address 127.5.22.62:38427)
0928 06:35:24.742506 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:24.742510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:24.743366 (+ 856us) server_negotiation.cc:407] Connection header received
0928 06:35:24.744857 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:24.744859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:24.744910 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:24.744991 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:24.746414 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:24.746961 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:24.748372 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:24.748765 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:24.748878 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:24.751584 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:24.751588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:24.751591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:24.751624 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:24.753132 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:24.753665 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:24.753671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:24.753673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:24.753757 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:24.754112 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:24.754117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:24.754119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:24.754305 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:24.754468 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:24.754994 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 06:35:24.755137 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260928 06:35:24.756120 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:24.742665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38427 (local address 127.5.22.1:44299)
0928 06:35:24.743183 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:24.743231 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:24.744627 (+ 1396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:24.745171 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:24.745179 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:24.745650 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:24.746183 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:24.746196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:24.747105 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:24.747108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:24.748239 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:24.748249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:24.748922 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:24.748925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:24.749088 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:24.749836 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:24.749861 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:24.751412 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:24.753259 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:24.753265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:24.753280 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:24.753546 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:24.753856 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:24.753859 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:24.753861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:24.754009 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:24.754474 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:24.754483 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:24.754756 (+ 273us) client_negotiation.cc:769] Sending connection context
0928 06:35:24.754954 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 06:35:24.755289 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":153,"threads_started":1}
WARNING: no policy specified for kudu/127.5.22.2@KRBTEST.COM; defaulting to no policy
I20260928 06:35:24.757515 7869 heartbeater.cc:347] Connected to a master server at 127.5.22.62:38427
I20260928 06:35:24.757766 7869 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:24.758267 7869 heartbeater.cc:511] Master 127.5.22.62:38427 requested a full tablet report, sending...
I20260928 06:35:24.759817 7677 ts_manager.cc:194] Registered new tserver with Master: 71b7e043cfa94b46b781df0b434501c4 (127.5.22.1:33931)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.2@KRBTEST.COM" created.
I20260928 06:35:24.761008 7677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.1@KRBTEST.COM'} at 127.5.22.1:44299
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
I20260928 06:35:24.777469 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:38427
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:24.893719 7874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:24.893996 7874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:24.894071 7874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:24.899060 7874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:24.899130 7874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:24.899232 7874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:35:24.905268 7874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:38427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:24.906490 7874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:24.907608 7874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:24.918322 7882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:24.918412 7879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:24.918314 7880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:24.918570 7874 server_base.cc:1034] running on GCE node
I20260928 06:35:24.919322 7874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:24.920095 7874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:24.921293 7874 hybrid_clock.cc:648] HybridClock initialized: now 1790577324921274 us; error 31 us; skew 500 ppm
Sep 28 06:35:24 dist-test-slave-j0qf krb5kdc[7632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577324, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:24.924263 7874 init.cc:377] Logged in from keytab as kudu/127.5.22.2@KRBTEST.COM (short username kudu)
I20260928 06:35:24.925227 7874 webserver.cc:460] Webserver started at http://127.5.22.2:35591/ using document root <none> and password file <none>
I20260928 06:35:24.925853 7874 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:24.925963 7874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:24.926222 7874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:24.928509 7874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "cb7676ecb1d041b6bbb516e66aeab0dc"
format_stamp: "Formatted at 2026-09-28 06:35:24 on dist-test-slave-j0qf"
server_key: "15063ec46cc86ad29ebc9645fc16cf56"
server_key_iv: "117649420cba180a3cdd197a6cf62506"
server_key_version: "encryptionkey@0"
I20260928 06:35:24.929162 7874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "cb7676ecb1d041b6bbb516e66aeab0dc"
format_stamp: "Formatted at 2026-09-28 06:35:24 on dist-test-slave-j0qf"
server_key: "15063ec46cc86ad29ebc9645fc16cf56"
server_key_iv: "117649420cba180a3cdd197a6cf62506"
server_key_version: "encryptionkey@0"
I20260928 06:35:24.933524 7874 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 06:35:24.936304 7890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:24.937675 7874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:35:24.937824 7874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "cb7676ecb1d041b6bbb516e66aeab0dc"
format_stamp: "Formatted at 2026-09-28 06:35:24 on dist-test-slave-j0qf"
server_key: "15063ec46cc86ad29ebc9645fc16cf56"
server_key_iv: "117649420cba180a3cdd197a6cf62506"
server_key_version: "encryptionkey@0"
I20260928 06:35:24.937955 7874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:24.960286 7874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:24.963966 7874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:24.964181 7874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:24.964779 7874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:24.965598 7874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:24.965667 7874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:24.965730 7874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:24.965781 7874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:24.976903 7874 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:42189
I20260928 06:35:24.977011 8004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:42189 every 8 connection(s)
I20260928 06:35:24.977860 7874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:24.984472 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 7874
I20260928 06:35:24.984571 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:24.984866 5208 external_mini_cluster.cc:1428] Setting key 3f2c14ee46e240f8b496bc6fd63ce57c
Sep 28 06:35:24 dist-test-slave-j0qf krb5kdc[7632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577324, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:24.992684 7717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:24.979719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:56359 (local address 127.5.22.62:38427)
0928 06:35:24.979883 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:24.979888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:24.981173 (+ 1285us) server_negotiation.cc:407] Connection header received
0928 06:35:24.982001 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:24.982004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:24.982051 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:24.982132 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:24.983335 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:24.983962 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:24.985218 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:24.985507 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:24.985631 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:24.988377 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:24.988381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:24.988383 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:24.988405 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:24.989935 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:24.990711 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:24.990716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:24.990718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:24.990826 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:24.991374 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:24.991377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:24.991380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:24.991637 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:24.991719 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:24.992443 (+ 724us) server_negotiation.cc:299] Negotiation successful
0928 06:35:24.992559 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 06:35:24.994041 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:24.980140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38427 (local address 127.5.22.2:56359)
0928 06:35:24.981016 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:24.981062 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:24.981818 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:24.982286 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:24.982293 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:24.982708 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:24.983173 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:24.983186 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:24.984077 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:24.984080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:24.985089 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:24.985098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:24.985681 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:24.985684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:24.985828 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:24.986548 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:24.986565 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:24.988211 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:24.990062 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:24.990070 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:24.990088 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:24.990500 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:24.990925 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:24.990931 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:24.990936 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:24.991176 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:24.991757 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:24.991766 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:24.992145 (+ 379us) client_negotiation.cc:769] Sending connection context
0928 06:35:24.992396 (+ 251us) client_negotiation.cc:240] Negotiation successful
0928 06:35:24.992814 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":539,"threads_started":1}
I20260928 06:35:24.995397 8005 heartbeater.cc:347] Connected to a master server at 127.5.22.62:38427
WARNING: no policy specified for kudu/127.5.22.3@KRBTEST.COM; defaulting to no policy
I20260928 06:35:24.995703 8005 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:24.996304 8005 heartbeater.cc:511] Master 127.5.22.62:38427 requested a full tablet report, sending...
I20260928 06:35:24.997339 7677 ts_manager.cc:194] Registered new tserver with Master: cb7676ecb1d041b6bbb516e66aeab0dc (127.5.22.2:42189)
I20260928 06:35:24.997874 7677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.2@KRBTEST.COM'} at 127.5.22.2:56359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
I20260928 06:35:25.018107 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:38427
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:25.131383 8010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:25.131644 8010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:25.131714 8010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:25.135840 8010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:25.135903 8010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:25.136014 8010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:35:25.141316 8010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:38427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:25.142473 8010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:25.143543 8010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:35:25.153385 8010 server_base.cc:1034] running on GCE node
W20260928 06:35:25.153350 8015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:25.153323 8018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:25.153329 8016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:25.153919 8010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:25.154578 8010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:25.155758 8010 hybrid_clock.cc:648] HybridClock initialized: now 1790577325155733 us; error 48 us; skew 500 ppm
Sep 28 06:35:25 dist-test-slave-j0qf krb5kdc[7632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577325, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:25.158447 8010 init.cc:377] Logged in from keytab as kudu/127.5.22.3@KRBTEST.COM (short username kudu)
I20260928 06:35:25.159299 8010 webserver.cc:460] Webserver started at http://127.5.22.3:44661/ using document root <none> and password file <none>
I20260928 06:35:25.159813 8010 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:25.159871 8010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:25.160032 8010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:25.162159 8010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "a122b1b8e257478c98dc1ba9d79ad56a"
format_stamp: "Formatted at 2026-09-28 06:35:25 on dist-test-slave-j0qf"
server_key: "ad1979ec3a6ec7777391e9407d0be9c1"
server_key_iv: "400fcb439614508b59a3eeb20dda2fbd"
server_key_version: "encryptionkey@0"
I20260928 06:35:25.162796 8010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "a122b1b8e257478c98dc1ba9d79ad56a"
format_stamp: "Formatted at 2026-09-28 06:35:25 on dist-test-slave-j0qf"
server_key: "ad1979ec3a6ec7777391e9407d0be9c1"
server_key_iv: "400fcb439614508b59a3eeb20dda2fbd"
server_key_version: "encryptionkey@0"
I20260928 06:35:25.166491 8010 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:35:25.168843 8025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:25.170040 8010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:25.170173 8010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "a122b1b8e257478c98dc1ba9d79ad56a"
format_stamp: "Formatted at 2026-09-28 06:35:25 on dist-test-slave-j0qf"
server_key: "ad1979ec3a6ec7777391e9407d0be9c1"
server_key_iv: "400fcb439614508b59a3eeb20dda2fbd"
server_key_version: "encryptionkey@0"
I20260928 06:35:25.170284 8010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:25.184561 8010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:25.187446 8010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:25.187611 8010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:25.188140 8010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:25.188906 8010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:25.188952 8010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:25.188990 8010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:25.189013 8010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:25.199859 8010 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:35675
I20260928 06:35:25.199877 8139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:35675 every 8 connection(s)
I20260928 06:35:25.200763 8010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:25.204741 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 8010
I20260928 06:35:25.204901 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:25.205171 5208 external_mini_cluster.cc:1428] Setting key 873353c61044ed5d59bbc36a5721c3eb
Sep 28 06:35:25 dist-test-slave-j0qf krb5kdc[7632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577325, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:25.214141 7717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:25.202636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:49479 (local address 127.5.22.62:38427)
0928 06:35:25.202772 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:25.202776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:25.203676 (+ 900us) server_negotiation.cc:407] Connection header received
0928 06:35:25.205091 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:25.205094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:25.205151 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:25.205262 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:25.206492 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:25.207026 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:25.208225 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:25.208448 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:25.208568 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:25.211036 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:25.211039 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:25.211042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:25.211065 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:25.212418 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:25.212872 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:25.212874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:25.212876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:25.212948 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:25.213230 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:25.213232 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:25.213233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:25.213391 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:25.213462 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:25.213933 (+ 471us) server_negotiation.cc:299] Negotiation successful
0928 06:35:25.214025 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:35:25.215083 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:25.203014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38427 (local address 127.5.22.3:49479)
0928 06:35:25.203526 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:25.203575 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:25.204862 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:25.205438 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:25.205445 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:25.205821 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:25.206294 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:25.206306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:25.207152 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:25.207155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:25.208102 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:25.208111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:25.208603 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:25.208608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:25.208782 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:25.209364 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:25.209380 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:25.210895 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:25.212521 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:25.212525 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:25.212536 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:25.212787 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:25.213025 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:25.213027 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:25.213028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:25.213152 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:25.213477 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:25.213481 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:25.213732 (+ 251us) client_negotiation.cc:769] Sending connection context
0928 06:35:25.213886 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 06:35:25.214129 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":174,"threads_started":1}
I20260928 06:35:25.216171 8140 heartbeater.cc:347] Connected to a master server at 127.5.22.62:38427
I20260928 06:35:25.216405 8140 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:25.216903 8140 heartbeater.cc:511] Master 127.5.22.62:38427 requested a full tablet report, sending...
I20260928 06:35:25.217988 7677 ts_manager.cc:194] Registered new tserver with Master: a122b1b8e257478c98dc1ba9d79ad56a (127.5.22.3:35675)
I20260928 06:35:25.218804 7677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.3@KRBTEST.COM'} at 127.5.22.3:49479
I20260928 06:35:25.219194 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 06:35:25.232326 7717 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:25.230808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33788 (local address 127.5.22.62:38427)
0928 06:35:25.230962 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:25.230968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:25.231222 (+ 254us) server_negotiation.cc:407] Connection header received
0928 06:35:25.231325 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:25.231328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:25.231370 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:25.231439 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:25.232187 (+ 748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33788: BlockingRecv error: recv got EOF from 127.0.0.1:33788 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:35:25.233495 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 7740
I20260928 06:35:25.239390 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 7874
I20260928 06:35:25.244498 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 8010
I20260928 06:35:25.249648 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 7645
2026-09-28T06:35:25Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2180 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:25 dist-test-slave-j0qf krb5kdc[8152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:25 dist-test-slave-j0qf krb5kdc[8152](info): set up 2 sockets
Sep 28 06:35:25 dist-test-slave-j0qf krb5kdc[8152](info): commencing operation
krb5kdc: starting...
W20260928 06:35:26.300756 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:35:26 dist-test-slave-j0qf krb5kdc[8152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:35:26.430650 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:33979
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:33979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:26.540023 8165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:26.540280 8165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:26.540321 8165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:26.544373 8165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:26.544435 8165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:26.544454 8165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:26.544471 8165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:26.544486 8165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:26.549726 8165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:33979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:33979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:26.550832 8165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:26.551867 8165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:26.560614 8170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:26.560640 8171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:26.560751 8165 server_base.cc:1034] running on GCE node
W20260928 06:35:26.560773 8174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:26.561452 8165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:26.562706 8165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:26.563898 8165 hybrid_clock.cc:648] HybridClock initialized: now 1790577326563883 us; error 31 us; skew 500 ppm
Sep 28 06:35:26 dist-test-slave-j0qf krb5kdc[8152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577326, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:26.569547 8165 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:35:26.570577 8165 webserver.cc:460] Webserver started at http://127.5.22.62:40973/ using document root <none> and password file <none>
I20260928 06:35:26.571163 8165 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:26.571261 8165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:26.571527 8165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:26.573613 8165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "bd2a15fc020148c6927885329e13b7f0"
format_stamp: "Formatted at 2026-09-28 06:35:26 on dist-test-slave-j0qf"
server_key: "345fd6393acc94047a981c5ed2da6638"
server_key_iv: "40e3e83a520f319eb62984957a4306e1"
server_key_version: "encryptionkey@0"
I20260928 06:35:26.574198 8165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "bd2a15fc020148c6927885329e13b7f0"
format_stamp: "Formatted at 2026-09-28 06:35:26 on dist-test-slave-j0qf"
server_key: "345fd6393acc94047a981c5ed2da6638"
server_key_iv: "40e3e83a520f319eb62984957a4306e1"
server_key_version: "encryptionkey@0"
I20260928 06:35:26.578063 8165 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 06:35:26.580578 8181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:26.581780 8165 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:26.581928 8165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "bd2a15fc020148c6927885329e13b7f0"
format_stamp: "Formatted at 2026-09-28 06:35:26 on dist-test-slave-j0qf"
server_key: "345fd6393acc94047a981c5ed2da6638"
server_key_iv: "40e3e83a520f319eb62984957a4306e1"
server_key_version: "encryptionkey@0"
I20260928 06:35:26.582041 8165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:26.603668 8165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:26.606786 8165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:26.607005 8165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:26.614428 8165 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:33979
I20260928 06:35:26.614706 8233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:33979 every 8 connection(s)
I20260928 06:35:26.615456 8165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:26.617101 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 8165
I20260928 06:35:26.617223 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:35:26.617522 5208 external_mini_cluster.cc:1428] Setting key 1e75fc1310e6be2e50b23674f8f04c12
I20260928 06:35:26.618831 8234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:26.625586 8234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd2a15fc020148c6927885329e13b7f0: Bootstrap starting.
Sep 28 06:35:26 dist-test-slave-j0qf krb5kdc[8152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:26.627991 8234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd2a15fc020148c6927885329e13b7f0: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:26.628732 8234 log.cc:824] T 00000000000000000000000000000000 P bd2a15fc020148c6927885329e13b7f0: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:26.630993 8234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd2a15fc020148c6927885329e13b7f0: No bootstrap required, opened a new log
I20260928 06:35:26.632262 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.618835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.5.22.62:33979)
0928 06:35:26.619327 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.619353 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.619390 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:35:26.620030 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.620046 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.620371 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.620659 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.621632 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.622685 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.623828 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.624282 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.624397 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.627261 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:26.627267 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:26.627279 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:26.627311 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:26.629195 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.629860 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.629864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.629869 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.629979 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.630277 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.630279 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.630280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.630739 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:26.630890 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.631229 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.631543 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":174,"threads_started":1}
I20260928 06:35:26.634761 8234 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd2a15fc020148c6927885329e13b7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2a15fc020148c6927885329e13b7f0" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 33979 } }
I20260928 06:35:26.635018 8234 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd2a15fc020148c6927885329e13b7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:26.635098 8234 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd2a15fc020148c6927885329e13b7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd2a15fc020148c6927885329e13b7f0, State: Initialized, Role: FOLLOWER
I20260928 06:35:26.635525 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:26.635551 8234 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd2a15fc020148c6927885329e13b7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2a15fc020148c6927885329e13b7f0" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 33979 } }
I20260928 06:35:26.635699 8234 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd2a15fc020148c6927885329e13b7f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:26.635757 8234 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd2a15fc020148c6927885329e13b7f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:26.635846 8234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd2a15fc020148c6927885329e13b7f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:26.637318 8234 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd2a15fc020148c6927885329e13b7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2a15fc020148c6927885329e13b7f0" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 33979 } }
I20260928 06:35:26.637727 8234 leader_election.cc:302] T 00000000000000000000000000000000 P bd2a15fc020148c6927885329e13b7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd2a15fc020148c6927885329e13b7f0; no voters:
I20260928 06:35:26.638114 8234 leader_election.cc:288] T 00000000000000000000000000000000 P bd2a15fc020148c6927885329e13b7f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:26.638190 8240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd2a15fc020148c6927885329e13b7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:26.638412 8240 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd2a15fc020148c6927885329e13b7f0 [term 1 LEADER]: Becoming Leader. State: Replica: bd2a15fc020148c6927885329e13b7f0, State: Running, Role: LEADER
I20260928 06:35:26.638720 8240 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd2a15fc020148c6927885329e13b7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2a15fc020148c6927885329e13b7f0" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 33979 } }
I20260928 06:35:26.639427 8234 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd2a15fc020148c6927885329e13b7f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:26.639904 8240 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd2a15fc020148c6927885329e13b7f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd2a15fc020148c6927885329e13b7f0. Latest consensus state: current_term: 1 leader_uuid: "bd2a15fc020148c6927885329e13b7f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2a15fc020148c6927885329e13b7f0" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 33979 } } }
I20260928 06:35:26.640019 8240 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd2a15fc020148c6927885329e13b7f0 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:26.640745 8241 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd2a15fc020148c6927885329e13b7f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd2a15fc020148c6927885329e13b7f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2a15fc020148c6927885329e13b7f0" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 33979 } } }
I20260928 06:35:26.640848 8241 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd2a15fc020148c6927885329e13b7f0 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:26.641126 8244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:26.644891 8244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:26.650934 8244 catalog_manager.cc:1347] Generated new cluster ID: 3a41171feec3405d8b2c7b918f17ab34
I20260928 06:35:26.651014 8244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:26.661022 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.655218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51958 (local address 127.5.22.62:33979)
0928 06:35:26.655353 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.655357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.655644 (+ 287us) server_negotiation.cc:407] Connection header received
0928 06:35:26.655778 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.655782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.655829 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.655933 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.656550 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.657337 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.658569 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.658929 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.659024 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.659985 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:26.659989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:26.659992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:26.660556 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:26.660588 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:26.660605 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:26.660694 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.660815 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.660872 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:35:26.669400 8244 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:26.670421 8244 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:26.678828 8244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd2a15fc020148c6927885329e13b7f0: Generated new TSK 0
I20260928 06:35:26.679816 8244 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:26.692757 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 8165
2026-09-28T06:35:26Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1441 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:26 dist-test-slave-j0qf krb5kdc[8267](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:26 dist-test-slave-j0qf krb5kdc[8267](info): set up 2 sockets
Sep 28 06:35:26 dist-test-slave-j0qf krb5kdc[8267](info): commencing operation
krb5kdc: starting...
W20260928 06:35:27.733279 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:35:27 dist-test-slave-j0qf krb5kdc[8267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:35:27.862484 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:39619
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:39619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:27.974539 8280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:27.974851 8280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:27.974953 8280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:27.979084 8280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:27.979163 8280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:27.979200 8280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:27.979256 8280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:27.979295 8280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:27.984869 8280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:39619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:39619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:27.985939 8280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:27.987011 8280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:35:27.995890 8280 server_base.cc:1034] running on GCE node
W20260928 06:35:27.995868 8286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:27.995851 8288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:27.995851 8285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:27.996718 8280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:27.997844 8280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:27.999059 8280 hybrid_clock.cc:648] HybridClock initialized: now 1790577327999031 us; error 45 us; skew 500 ppm
Sep 28 06:35:27 dist-test-slave-j0qf krb5kdc[8267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577327, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:28.001953 8280 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:35:28.002954 8280 webserver.cc:460] Webserver started at http://127.5.22.62:34603/ using document root <none> and password file <none>
I20260928 06:35:28.003515 8280 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:28.003607 8280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:28.003841 8280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:28.005877 8280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "e1dc347583c545af996122682ce06289"
format_stamp: "Formatted at 2026-09-28 06:35:27 on dist-test-slave-j0qf"
server_key: "c3b1c78d892691eef8bd33626b2dadaa"
server_key_iv: "47f2cf7af64b44eb68eed6252c919159"
server_key_version: "encryptionkey@0"
I20260928 06:35:28.006512 8280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "e1dc347583c545af996122682ce06289"
format_stamp: "Formatted at 2026-09-28 06:35:27 on dist-test-slave-j0qf"
server_key: "c3b1c78d892691eef8bd33626b2dadaa"
server_key_iv: "47f2cf7af64b44eb68eed6252c919159"
server_key_version: "encryptionkey@0"
I20260928 06:35:28.010507 8280 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 06:35:28.013064 8295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:28.014364 8280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:28.014510 8280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "e1dc347583c545af996122682ce06289"
format_stamp: "Formatted at 2026-09-28 06:35:27 on dist-test-slave-j0qf"
server_key: "c3b1c78d892691eef8bd33626b2dadaa"
server_key_iv: "47f2cf7af64b44eb68eed6252c919159"
server_key_version: "encryptionkey@0"
I20260928 06:35:28.014628 8280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:28.030194 8280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:28.033604 8280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:28.033814 8280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:28.041083 8280 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:39619
I20260928 06:35:28.041106 8348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:39619 every 8 connection(s)
I20260928 06:35:28.042052 8280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:28.044823 8349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:28.049021 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 8280
I20260928 06:35:28.049116 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:35:28.049379 5208 external_mini_cluster.cc:1428] Setting key e99beda7a30cbbc4d297194841078780
I20260928 06:35:28.050598 8349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1dc347583c545af996122682ce06289: Bootstrap starting.
I20260928 06:35:28.053355 8349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1dc347583c545af996122682ce06289: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:28.054214 8349 log.cc:824] T 00000000000000000000000000000000 P e1dc347583c545af996122682ce06289: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:28.056275 8349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1dc347583c545af996122682ce06289: No bootstrap required, opened a new log
Sep 28 06:35:28 dist-test-slave-j0qf krb5kdc[8267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:28.060402 8349 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1dc347583c545af996122682ce06289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1dc347583c545af996122682ce06289" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 39619 } }
I20260928 06:35:28.060707 8349 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1dc347583c545af996122682ce06289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:28.060779 8349 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1dc347583c545af996122682ce06289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dc347583c545af996122682ce06289, State: Initialized, Role: FOLLOWER
I20260928 06:35:28.061331 8349 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1dc347583c545af996122682ce06289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1dc347583c545af996122682ce06289" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 39619 } }
I20260928 06:35:28.061501 8349 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1dc347583c545af996122682ce06289 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:28.061576 8349 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1dc347583c545af996122682ce06289 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:28.061681 8349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1dc347583c545af996122682ce06289 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:28.062830 8349 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1dc347583c545af996122682ce06289 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1dc347583c545af996122682ce06289" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 39619 } }
I20260928 06:35:28.063185 8349 leader_election.cc:302] T 00000000000000000000000000000000 P e1dc347583c545af996122682ce06289 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1dc347583c545af996122682ce06289; no voters:
I20260928 06:35:28.063561 8349 leader_election.cc:288] T 00000000000000000000000000000000 P e1dc347583c545af996122682ce06289 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:28.063692 8354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1dc347583c545af996122682ce06289 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:28.063936 8354 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1dc347583c545af996122682ce06289 [term 1 LEADER]: Becoming Leader. State: Replica: e1dc347583c545af996122682ce06289, State: Running, Role: LEADER
I20260928 06:35:28.064272 8354 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1dc347583c545af996122682ce06289 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1dc347583c545af996122682ce06289" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 39619 } }
I20260928 06:35:28.064852 8349 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1dc347583c545af996122682ce06289 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:28.065022 8352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:28.050883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49796 (local address 127.5.22.62:39619)
0928 06:35:28.051312 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:28.051331 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:28.051374 (+ 43us) server_negotiation.cc:407] Connection header received
0928 06:35:28.052237 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:28.052259 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:28.052715 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:28.053076 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:28.053838 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:28.055242 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:28.056071 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:28.056758 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:28.056921 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:28.059669 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:28.059675 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:28.059692 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:28.059723 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:28.061756 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:28.062165 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:28.062171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:28.062177 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:28.062345 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:28.062597 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:28.062601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:28.062603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:28.063129 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:28.063329 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:28.063615 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 06:35:28.064073 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":121,"threads_started":1}
I20260928 06:35:28.065366 8354 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1dc347583c545af996122682ce06289 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1dc347583c545af996122682ce06289. Latest consensus state: current_term: 1 leader_uuid: "e1dc347583c545af996122682ce06289" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1dc347583c545af996122682ce06289" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 39619 } } }
I20260928 06:35:28.065565 8354 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1dc347583c545af996122682ce06289 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:28.065780 8355 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1dc347583c545af996122682ce06289 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1dc347583c545af996122682ce06289" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1dc347583c545af996122682ce06289" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 39619 } } }
I20260928 06:35:28.065918 8355 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1dc347583c545af996122682ce06289 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:28.066015 8360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:28.068845 8360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:28.074465 8360 catalog_manager.cc:1347] Generated new cluster ID: fdbd27f41e3043409848270f11f541ad
I20260928 06:35:28.074539 8360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:28.092531 8360 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:28.093603 8360 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:28.099463 8360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1dc347583c545af996122682ce06289: Generated new TSK 0
I20260928 06:35:28.100191 8360 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.22.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
I20260928 06:35:28.131127 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:39619
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:28.245409 8375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:28.245735 8375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:28.245814 8375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:28.249946 8375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:28.250053 8375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:28.250146 8375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:35:28.255528 8375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:39619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:28.256690 8375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:28.257745 8375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:35:28.268232 8375 server_base.cc:1034] running on GCE node
W20260928 06:35:28.268189 8383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:28.268194 8380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:28.268194 8381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:28.268890 8375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:28.269490 8375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:28.270646 8375 hybrid_clock.cc:648] HybridClock initialized: now 1790577328270633 us; error 27 us; skew 500 ppm
Sep 28 06:35:28 dist-test-slave-j0qf krb5kdc[8267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577328, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:28.273432 8375 init.cc:377] Logged in from keytab as kudu/127.5.22.1@KRBTEST.COM (short username kudu)
I20260928 06:35:28.274372 8375 webserver.cc:460] Webserver started at http://127.5.22.1:36809/ using document root <none> and password file <none>
I20260928 06:35:28.274945 8375 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:28.275007 8375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:28.275251 8375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:28.277302 8375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "2ccaa6b6423e410ba2aa8d5ee068c317"
format_stamp: "Formatted at 2026-09-28 06:35:28 on dist-test-slave-j0qf"
server_key: "1ef406885d8f0f0ad1cc746184cd8941"
server_key_iv: "946f34d60d555d3d18da2740fe7ea030"
server_key_version: "encryptionkey@0"
I20260928 06:35:28.277872 8375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "2ccaa6b6423e410ba2aa8d5ee068c317"
format_stamp: "Formatted at 2026-09-28 06:35:28 on dist-test-slave-j0qf"
server_key: "1ef406885d8f0f0ad1cc746184cd8941"
server_key_iv: "946f34d60d555d3d18da2740fe7ea030"
server_key_version: "encryptionkey@0"
I20260928 06:35:28.281833 8375 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 06:35:28.284400 8390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:28.285678 8375 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 06:35:28.285773 8375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "2ccaa6b6423e410ba2aa8d5ee068c317"
format_stamp: "Formatted at 2026-09-28 06:35:28 on dist-test-slave-j0qf"
server_key: "1ef406885d8f0f0ad1cc746184cd8941"
server_key_iv: "946f34d60d555d3d18da2740fe7ea030"
server_key_version: "encryptionkey@0"
I20260928 06:35:28.285907 8375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:28.307274 8375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:28.310743 8375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:28.310962 8375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:28.311586 8375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:28.312450 8375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:28.312520 8375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:28.312598 8375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:28.312650 8375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:28.321995 8375 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:39301
I20260928 06:35:28.322082 8503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:39301 every 8 connection(s)
I20260928 06:35:28.323019 8375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:28.328284 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 8375
I20260928 06:35:28.328439 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:28.328722 5208 external_mini_cluster.cc:1428] Setting key 34de2ca277a52520fbe65e4baee7a36b
Sep 28 06:35:28 dist-test-slave-j0qf krb5kdc[8267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577328, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.22.2@KRBTEST.COM; defaulting to no policy
I20260928 06:35:28.338153 8352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:28.325017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:50433 (local address 127.5.22.62:39619)
0928 06:35:28.325167 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:28.325171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:28.326002 (+ 831us) server_negotiation.cc:407] Connection header received
0928 06:35:28.327524 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:28.327527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:28.327629 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:28.327736 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:28.329075 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:28.329750 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:28.331409 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:28.331647 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:28.331734 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:28.334547 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:28.334550 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:28.334552 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:28.334576 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:28.336045 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:28.336585 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:28.336588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:28.336590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:28.336671 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:28.337024 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:28.337027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:28.337029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:28.337217 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:28.337325 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:28.337852 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 06:35:28.337979 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 06:35:28.339017 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:28.325405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39619 (local address 127.5.22.1:50433)
0928 06:35:28.325851 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:28.325895 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:28.327267 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:28.327914 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:28.327921 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:28.328313 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:28.328929 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:28.328945 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:28.329884 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:28.329888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:28.331282 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:28.331296 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:28.331812 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:28.331819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:28.332070 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:28.332753 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:28.332769 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:28.334357 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:28.336185 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:28.336191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:28.336206 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:28.336470 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:28.336771 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:28.336774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:28.336776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:28.336922 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:28.337325 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:28.337332 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:28.337593 (+ 261us) client_negotiation.cc:769] Sending connection context
0928 06:35:28.337778 (+ 185us) client_negotiation.cc:240] Negotiation successful
0928 06:35:28.338129 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":114,"threads_started":1}
I20260928 06:35:28.340369 8504 heartbeater.cc:347] Connected to a master server at 127.5.22.62:39619
I20260928 06:35:28.340639 8504 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:28.341207 8504 heartbeater.cc:511] Master 127.5.22.62:39619 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.2@KRBTEST.COM" created.
I20260928 06:35:28.342890 8312 ts_manager.cc:194] Registered new tserver with Master: 2ccaa6b6423e410ba2aa8d5ee068c317 (127.5.22.1:39301)
I20260928 06:35:28.344551 8312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.1@KRBTEST.COM'} at 127.5.22.1:50433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
I20260928 06:35:28.357694 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:39619
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:28.471338 8510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:28.471742 8510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:28.471837 8510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:28.477404 8510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:28.477468 8510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:28.477528 8510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:35:28.483101 8510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:39619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:28.484102 8510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:28.485070 8510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:28.494872 8518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:28.494956 8510 server_base.cc:1034] running on GCE node
W20260928 06:35:28.494872 8515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:28.494937 8516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:28.495554 8510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:28.496196 8510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:28.497400 8510 hybrid_clock.cc:648] HybridClock initialized: now 1790577328497388 us; error 30 us; skew 500 ppm
Sep 28 06:35:28 dist-test-slave-j0qf krb5kdc[8267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577328, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:28.500200 8510 init.cc:377] Logged in from keytab as kudu/127.5.22.2@KRBTEST.COM (short username kudu)
I20260928 06:35:28.500967 8510 webserver.cc:460] Webserver started at http://127.5.22.2:33665/ using document root <none> and password file <none>
I20260928 06:35:28.501470 8510 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:28.501526 8510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:28.501688 8510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:28.503644 8510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "d9e33c6b9b4946919e9ce133843798a7"
format_stamp: "Formatted at 2026-09-28 06:35:28 on dist-test-slave-j0qf"
server_key: "b5966ceb18c4a6d4f5bd3ffb2d595952"
server_key_iv: "a7ad77e24e63f6592cf4cb5568a7c13e"
server_key_version: "encryptionkey@0"
I20260928 06:35:28.504211 8510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "d9e33c6b9b4946919e9ce133843798a7"
format_stamp: "Formatted at 2026-09-28 06:35:28 on dist-test-slave-j0qf"
server_key: "b5966ceb18c4a6d4f5bd3ffb2d595952"
server_key_iv: "a7ad77e24e63f6592cf4cb5568a7c13e"
server_key_version: "encryptionkey@0"
I20260928 06:35:28.507787 8510 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:35:28.510061 8525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:28.511328 8510 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:35:28.511420 8510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "d9e33c6b9b4946919e9ce133843798a7"
format_stamp: "Formatted at 2026-09-28 06:35:28 on dist-test-slave-j0qf"
server_key: "b5966ceb18c4a6d4f5bd3ffb2d595952"
server_key_iv: "a7ad77e24e63f6592cf4cb5568a7c13e"
server_key_version: "encryptionkey@0"
I20260928 06:35:28.511497 8510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:28.532483 8510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:28.535854 8510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:28.536073 8510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:28.536675 8510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:28.537513 8510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:28.537606 8510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:28.537678 8510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:28.537724 8510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:28.546675 8510 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:36175
I20260928 06:35:28.546705 8639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:36175 every 8 connection(s)
I20260928 06:35:28.547570 8510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:28.554888 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 8510
I20260928 06:35:28.555023 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:28.555327 5208 external_mini_cluster.cc:1428] Setting key 9fbc46c132ee8cfedf9715d107737378
Sep 28 06:35:28 dist-test-slave-j0qf krb5kdc[8267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577328, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:28.561600 8352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:28.549379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:48677 (local address 127.5.22.62:39619)
0928 06:35:28.549532 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:28.549537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:28.550424 (+ 887us) server_negotiation.cc:407] Connection header received
0928 06:35:28.551778 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:28.551780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:28.551826 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:28.551899 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:28.553052 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:28.553571 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:28.554762 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:28.555148 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:28.555254 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:28.557723 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:28.557726 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:28.557728 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:28.557750 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:28.559295 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:28.559935 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:28.559937 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:28.559939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:28.560018 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:28.560400 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:28.560402 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:28.560403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:28.560587 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:28.560665 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:28.561371 (+ 706us) server_negotiation.cc:299] Negotiation successful
0928 06:35:28.561475 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 06:35:28.562980 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:28.549825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39619 (local address 127.5.22.2:48677)
0928 06:35:28.550246 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:28.550289 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:28.551588 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:28.552078 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:28.552084 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:28.552446 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:28.552922 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:28.552935 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:28.553679 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:28.553682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:28.554642 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:28.554651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:28.555310 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:28.555313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:28.555482 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:28.556106 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:28.556122 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:28.557548 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:28.559425 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:28.559434 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:28.559452 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:28.559821 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:28.560108 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:28.560114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:28.560119 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:28.560307 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:28.560697 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:28.560707 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:28.561065 (+ 358us) client_negotiation.cc:769] Sending connection context
0928 06:35:28.561328 (+ 263us) client_negotiation.cc:240] Negotiation successful
0928 06:35:28.561714 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":98,"threads_started":1}
I20260928 06:35:28.564338 8640 heartbeater.cc:347] Connected to a master server at 127.5.22.62:39619
I20260928 06:35:28.564617 8640 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:28.565119 8640 heartbeater.cc:511] Master 127.5.22.62:39619 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.22.3@KRBTEST.COM; defaulting to no policy
I20260928 06:35:28.566397 8312 ts_manager.cc:194] Registered new tserver with Master: d9e33c6b9b4946919e9ce133843798a7 (127.5.22.2:36175)
I20260928 06:35:28.567001 8312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.2@KRBTEST.COM'} at 127.5.22.2:48677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
I20260928 06:35:28.585487 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:39619
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:28.698717 8645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:28.698988 8645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:28.699033 8645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:28.703070 8645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:28.703133 8645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:28.703193 8645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:35:28.708626 8645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:39619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:28.709723 8645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:28.710820 8645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:28.720839 8653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:28.720865 8651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:28.720862 8650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:28.721143 8645 server_base.cc:1034] running on GCE node
I20260928 06:35:28.721601 8645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:28.722172 8645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:28.723335 8645 hybrid_clock.cc:648] HybridClock initialized: now 1790577328723316 us; error 34 us; skew 500 ppm
Sep 28 06:35:28 dist-test-slave-j0qf krb5kdc[8267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577328, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:28.726063 8645 init.cc:377] Logged in from keytab as kudu/127.5.22.3@KRBTEST.COM (short username kudu)
I20260928 06:35:28.726948 8645 webserver.cc:460] Webserver started at http://127.5.22.3:41349/ using document root <none> and password file <none>
I20260928 06:35:28.727507 8645 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:28.727622 8645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:28.727852 8645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:28.729903 8645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "ef94e2183f344e18bdb74cb198f3161a"
format_stamp: "Formatted at 2026-09-28 06:35:28 on dist-test-slave-j0qf"
server_key: "db3468b8822c8b808c7c653e75683f51"
server_key_iv: "2e19160794313861a3376c51c69e9b19"
server_key_version: "encryptionkey@0"
I20260928 06:35:28.730532 8645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "ef94e2183f344e18bdb74cb198f3161a"
format_stamp: "Formatted at 2026-09-28 06:35:28 on dist-test-slave-j0qf"
server_key: "db3468b8822c8b808c7c653e75683f51"
server_key_iv: "2e19160794313861a3376c51c69e9b19"
server_key_version: "encryptionkey@0"
I20260928 06:35:28.734573 8645 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 06:35:28.737133 8660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:28.738699 8645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:28.738863 8645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "ef94e2183f344e18bdb74cb198f3161a"
format_stamp: "Formatted at 2026-09-28 06:35:28 on dist-test-slave-j0qf"
server_key: "db3468b8822c8b808c7c653e75683f51"
server_key_iv: "2e19160794313861a3376c51c69e9b19"
server_key_version: "encryptionkey@0"
I20260928 06:35:28.739032 8645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:28.754698 8645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:28.758626 8645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:28.758860 8645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:28.759541 8645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:28.760452 8645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:28.760531 8645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:28.760608 8645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:28.760663 8645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:28.771014 8645 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:46097
I20260928 06:35:28.771034 8774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:46097 every 8 connection(s)
I20260928 06:35:28.772032 8645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:28.782040 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 8645
I20260928 06:35:28.782205 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
Sep 28 06:35:28 dist-test-slave-j0qf krb5kdc[8267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577328, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:28.782640 5208 external_mini_cluster.cc:1428] Setting key f11e4292a806a1aaa6564f145f42157b
I20260928 06:35:28.787256 8352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:28.774123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:56385 (local address 127.5.22.62:39619)
0928 06:35:28.774304 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:28.774309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:28.775164 (+ 855us) server_negotiation.cc:407] Connection header received
0928 06:35:28.776661 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:28.776663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:28.776711 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:28.776791 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:28.778072 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:28.778734 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:28.779995 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:28.780220 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:28.780307 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:28.783251 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:28.783255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:28.783257 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:28.783286 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:28.785206 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:28.785729 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:28.785731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:28.785732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:28.785807 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:28.786223 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:28.786227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:28.786228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:28.786473 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:28.786574 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:28.787037 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 06:35:28.787137 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 06:35:28.788159 8777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:28.774501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39619 (local address 127.5.22.3:56385)
0928 06:35:28.774994 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:28.775044 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:28.776433 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:28.776993 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:28.777000 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:28.777391 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:28.777898 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:28.777913 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:28.778861 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:28.778864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:28.779862 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:28.779872 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:28.780367 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:28.780370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:28.780521 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:28.781221 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:28.781240 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:28.783099 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:28.785334 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:28.785338 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:28.785351 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:28.785640 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:28.785941 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:28.785944 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:28.785945 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:28.786104 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:28.786584 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:28.786590 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:28.786827 (+ 237us) client_negotiation.cc:769] Sending connection context
0928 06:35:28.786992 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 06:35:28.787275 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":109,"threads_started":1}
I20260928 06:35:28.789356 8775 heartbeater.cc:347] Connected to a master server at 127.5.22.62:39619
I20260928 06:35:28.789675 8775 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:28.790180 8775 heartbeater.cc:511] Master 127.5.22.62:39619 requested a full tablet report, sending...
I20260928 06:35:28.791265 8312 ts_manager.cc:194] Registered new tserver with Master: ef94e2183f344e18bdb74cb198f3161a (127.5.22.3:46097)
I20260928 06:35:28.791862 8312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.3@KRBTEST.COM'} at 127.5.22.3:56385
I20260928 06:35:28.796730 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:35:28 dist-test-slave-j0qf krb5kdc[8267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577328, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 06:35:28 dist-test-slave-j0qf krb5kdc[8267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577328, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:28.822988 8352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:28.814241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56290 (local address 127.5.22.62:39619)
0928 06:35:28.814555 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:28.814560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:28.814602 (+ 42us) server_negotiation.cc:407] Connection header received
0928 06:35:28.814696 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:28.814699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:28.814742 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:28.814819 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:28.815519 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:28.816121 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:28.816965 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:28.817179 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:28.817258 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:28.819846 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:28.819849 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:28.819851 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:28.819876 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:28.821141 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:28.821638 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:28.821640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:28.821642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:28.821715 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:28.821957 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:28.821959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:28.821960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:28.822183 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:28.822287 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:28.822655 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 06:35:28.822793 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
W20260928 06:35:28.823438 8312 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56290
I20260928 06:35:28.824348 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 8375
I20260928 06:35:28.831955 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 8510
I20260928 06:35:28.837329 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 8645
I20260928 06:35:28.842911 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 8280
2026-09-28T06:35:28Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2152 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:28 dist-test-slave-j0qf krb5kdc[8787](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:28 dist-test-slave-j0qf krb5kdc[8787](info): set up 2 sockets
Sep 28 06:35:28 dist-test-slave-j0qf krb5kdc[8787](info): commencing operation
krb5kdc: starting...
W20260928 06:35:29.889008 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:35:29 dist-test-slave-j0qf krb5kdc[8787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:35:30.021891 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:45945
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:38349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:45945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:30.144580 8800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:30.144865 8800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:30.144910 8800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:30.149566 8800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:30.149639 8800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:30.149662 8800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:30.149681 8800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:30.149727 8800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:30.155845 8800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:38349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:45945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:45945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:30.157045 8800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:30.158231 8800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:30.167217 8805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:30.167220 8808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:30.167498 8800 server_base.cc:1034] running on GCE node
W20260928 06:35:30.167218 8806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:30.168263 8800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:30.169448 8800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:30.170643 8800 hybrid_clock.cc:648] HybridClock initialized: now 1790577330170623 us; error 32 us; skew 500 ppm
Sep 28 06:35:30 dist-test-slave-j0qf krb5kdc[8787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577330, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:30.173765 8800 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:35:30.174749 8800 webserver.cc:460] Webserver started at http://127.5.22.62:33803/ using document root <none> and password file <none>
I20260928 06:35:30.175336 8800 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:30.175432 8800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:30.175670 8800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:30.177770 8800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "6be98744056f43c7a28c6e5b1f4339cd"
format_stamp: "Formatted at 2026-09-28 06:35:30 on dist-test-slave-j0qf"
server_key: "b88b2c02fc7c21e1d0b3a976f9b10f87"
server_key_iv: "b9416d6a3802353512dd076f29ee0b1b"
server_key_version: "encryptionkey@0"
I20260928 06:35:30.178417 8800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "6be98744056f43c7a28c6e5b1f4339cd"
format_stamp: "Formatted at 2026-09-28 06:35:30 on dist-test-slave-j0qf"
server_key: "b88b2c02fc7c21e1d0b3a976f9b10f87"
server_key_iv: "b9416d6a3802353512dd076f29ee0b1b"
server_key_version: "encryptionkey@0"
I20260928 06:35:30.182265 8800 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 06:35:30.184947 8815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:30.186371 8800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:30.186517 8800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "6be98744056f43c7a28c6e5b1f4339cd"
format_stamp: "Formatted at 2026-09-28 06:35:30 on dist-test-slave-j0qf"
server_key: "b88b2c02fc7c21e1d0b3a976f9b10f87"
server_key_iv: "b9416d6a3802353512dd076f29ee0b1b"
server_key_version: "encryptionkey@0"
I20260928 06:35:30.186642 8800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:30.204360 8800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:30.207543 8800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:30.207758 8800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:30.214998 8800 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:45945
I20260928 06:35:30.215018 8867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:45945 every 8 connection(s)
I20260928 06:35:30.215973 8800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:30.218988 8869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:30.219045 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 8800
I20260928 06:35:30.219158 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:35:30.219466 5208 external_mini_cluster.cc:1428] Setting key 92a10628d6560bcbfa99835cd39b25ad
I20260928 06:35:30.226011 8869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6be98744056f43c7a28c6e5b1f4339cd: Bootstrap starting.
I20260928 06:35:30.228392 8869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6be98744056f43c7a28c6e5b1f4339cd: Neither blocks nor log segments found. Creating new log.
Sep 28 06:35:30 dist-test-slave-j0qf krb5kdc[8787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:30.229316 8869 log.cc:824] T 00000000000000000000000000000000 P 6be98744056f43c7a28c6e5b1f4339cd: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:30.231489 8869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6be98744056f43c7a28c6e5b1f4339cd: No bootstrap required, opened a new log
I20260928 06:35:30.234547 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:30.220948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37886 (local address 127.5.22.62:45945)
0928 06:35:30.221381 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:30.221401 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:30.221444 (+ 43us) server_negotiation.cc:407] Connection header received
0928 06:35:30.222264 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:30.222287 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:30.222745 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:30.223115 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:30.223785 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:30.225302 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:30.226207 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:30.226775 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:30.226956 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:30.229683 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:30.229688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:30.229723 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:30.229759 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:30.231529 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:30.232038 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:30.232042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:30.232062 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:30.232193 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:30.232525 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:30.232527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:30.232528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:30.232923 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:30.233089 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:30.233378 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 06:35:30.233687 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":113,"threads_started":1}
I20260928 06:35:30.235694 8869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6be98744056f43c7a28c6e5b1f4339cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6be98744056f43c7a28c6e5b1f4339cd" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 45945 } }
I20260928 06:35:30.235966 8869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6be98744056f43c7a28c6e5b1f4339cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:30.236014 8869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6be98744056f43c7a28c6e5b1f4339cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6be98744056f43c7a28c6e5b1f4339cd, State: Initialized, Role: FOLLOWER
I20260928 06:35:30.236474 8869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6be98744056f43c7a28c6e5b1f4339cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6be98744056f43c7a28c6e5b1f4339cd" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 45945 } }
I20260928 06:35:30.236629 8869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6be98744056f43c7a28c6e5b1f4339cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:30.236724 8869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6be98744056f43c7a28c6e5b1f4339cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:30.236851 8869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6be98744056f43c7a28c6e5b1f4339cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:30.237802 8869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6be98744056f43c7a28c6e5b1f4339cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6be98744056f43c7a28c6e5b1f4339cd" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 45945 } }
I20260928 06:35:30.238145 8869 leader_election.cc:302] T 00000000000000000000000000000000 P 6be98744056f43c7a28c6e5b1f4339cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6be98744056f43c7a28c6e5b1f4339cd; no voters:
I20260928 06:35:30.238534 8869 leader_election.cc:288] T 00000000000000000000000000000000 P 6be98744056f43c7a28c6e5b1f4339cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:30.238689 8874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6be98744056f43c7a28c6e5b1f4339cd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:30.238967 8874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6be98744056f43c7a28c6e5b1f4339cd [term 1 LEADER]: Becoming Leader. State: Replica: 6be98744056f43c7a28c6e5b1f4339cd, State: Running, Role: LEADER
I20260928 06:35:30.239338 8874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6be98744056f43c7a28c6e5b1f4339cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6be98744056f43c7a28c6e5b1f4339cd" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 45945 } }
I20260928 06:35:30.239811 8869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6be98744056f43c7a28c6e5b1f4339cd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:30.241083 8875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6be98744056f43c7a28c6e5b1f4339cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6be98744056f43c7a28c6e5b1f4339cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6be98744056f43c7a28c6e5b1f4339cd" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 45945 } } }
I20260928 06:35:30.241132 8876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6be98744056f43c7a28c6e5b1f4339cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6be98744056f43c7a28c6e5b1f4339cd. Latest consensus state: current_term: 1 leader_uuid: "6be98744056f43c7a28c6e5b1f4339cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6be98744056f43c7a28c6e5b1f4339cd" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 45945 } } }
I20260928 06:35:30.241236 8876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6be98744056f43c7a28c6e5b1f4339cd [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:30.241236 8875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6be98744056f43c7a28c6e5b1f4339cd [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:30.241673 8883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:30.244999 8883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:30.251039 8883 catalog_manager.cc:1347] Generated new cluster ID: ac99a122ef0d4a5a936b8cfc7a59cf7a
I20260928 06:35:30.251120 8883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:30.262811 8883 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:30.263808 8883 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:30.271296 8883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6be98744056f43c7a28c6e5b1f4339cd: Generated new TSK 0
I20260928 06:35:30.272039 8883 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.22.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
I20260928 06:35:30.309881 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:45945
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:38349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:30.420903 8895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:30.421207 8895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:30.421283 8895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:30.425422 8895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:30.425516 8895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:30.425616 8895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:35:30.430915 8895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:38349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:45945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:30.431967 8895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:30.433030 8895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:30.442816 8903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:30.442822 8901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:30.443042 8895 server_base.cc:1034] running on GCE node
W20260928 06:35:30.442828 8900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:30.443486 8895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:30.444113 8895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:30.445302 8895 hybrid_clock.cc:648] HybridClock initialized: now 1790577330445278 us; error 44 us; skew 500 ppm
Sep 28 06:35:30 dist-test-slave-j0qf krb5kdc[8787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577330, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:30.448005 8895 init.cc:377] Logged in from keytab as kudu/127.5.22.1@KRBTEST.COM (short username kudu)
I20260928 06:35:30.448887 8895 webserver.cc:460] Webserver started at http://127.5.22.1:35379/ using document root <none> and password file <none>
I20260928 06:35:30.449441 8895 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:30.449533 8895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:30.449767 8895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:30.451850 8895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "a69ae7b5925e4b61b05aafad3b3134f6"
format_stamp: "Formatted at 2026-09-28 06:35:30 on dist-test-slave-j0qf"
server_key: "17927949cea36250854ff32c447d4fd2"
server_key_iv: "0704b21dc008e7a0d5454089f66144e3"
server_key_version: "encryptionkey@0"
I20260928 06:35:30.452414 8895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "a69ae7b5925e4b61b05aafad3b3134f6"
format_stamp: "Formatted at 2026-09-28 06:35:30 on dist-test-slave-j0qf"
server_key: "17927949cea36250854ff32c447d4fd2"
server_key_iv: "0704b21dc008e7a0d5454089f66144e3"
server_key_version: "encryptionkey@0"
I20260928 06:35:30.456080 8895 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 06:35:30.458472 8910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:30.459767 8895 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:35:30.459900 8895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "a69ae7b5925e4b61b05aafad3b3134f6"
format_stamp: "Formatted at 2026-09-28 06:35:30 on dist-test-slave-j0qf"
server_key: "17927949cea36250854ff32c447d4fd2"
server_key_iv: "0704b21dc008e7a0d5454089f66144e3"
server_key_version: "encryptionkey@0"
I20260928 06:35:30.460009 8895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:30.495124 8895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:30.498220 8895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:30.498461 8895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:30.499096 8895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:30.499984 8895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:30.500057 8895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:30.500126 8895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:30.500170 8895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:30.508984 8895 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:38777
I20260928 06:35:30.509008 9024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:38777 every 8 connection(s)
I20260928 06:35:30.509877 8895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:30.516892 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 8895
I20260928 06:35:30.517020 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:30.517310 5208 external_mini_cluster.cc:1428] Setting key 3db85363e489487aaf65d9066e5765f8
Sep 28 06:35:30 dist-test-slave-j0qf krb5kdc[8787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577330, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:30.524745 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:30.511749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:39427 (local address 127.5.22.62:45945)
0928 06:35:30.511892 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:30.511896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:30.512706 (+ 810us) server_negotiation.cc:407] Connection header received
0928 06:35:30.514163 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:30.514167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:30.514227 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:30.514348 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:30.515480 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:30.516022 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:30.517482 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:30.517759 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:30.517888 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:30.520814 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:30.520820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:30.520825 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:30.520858 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:30.522592 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:30.523073 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:30.523080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:30.523084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:30.523182 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:30.523539 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:30.523543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:30.523545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:30.523787 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:30.523900 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:30.524370 (+ 470us) server_negotiation.cc:299] Negotiation successful
0928 06:35:30.524517 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 06:35:30.525354 9027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:30.512108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:45945 (local address 127.5.22.1:39427)
0928 06:35:30.512550 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:30.512597 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:30.513918 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:30.514527 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:30.514534 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:30.514875 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:30.515337 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:30.515349 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:30.516153 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:30.516159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:30.517322 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:30.517335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:30.517946 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:30.517950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:30.518134 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:30.518816 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:30.518843 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:30.520648 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:30.522702 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:30.522706 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:30.522717 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:30.522959 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:30.523307 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:30.523310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:30.523311 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:30.523439 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:30.523893 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:30.523898 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:30.524125 (+ 227us) client_negotiation.cc:769] Sending connection context
0928 06:35:30.524277 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 06:35:30.524515 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":114,"threads_started":1}
I20260928 06:35:30.526831 9025 heartbeater.cc:347] Connected to a master server at 127.5.22.62:45945
I20260928 06:35:30.527099 9025 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:30.527585 9025 heartbeater.cc:511] Master 127.5.22.62:45945 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.22.2@KRBTEST.COM; defaulting to no policy
I20260928 06:35:30.529186 8832 ts_manager.cc:194] Registered new tserver with Master: a69ae7b5925e4b61b05aafad3b3134f6 (127.5.22.1:38777)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.2@KRBTEST.COM" created.
I20260928 06:35:30.530560 8832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.1@KRBTEST.COM'} at 127.5.22.1:39427
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
I20260928 06:35:30.545529 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:45945
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:38349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:30.656291 9030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:30.656569 9030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:30.656612 9030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:30.661350 9030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:30.661415 9030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:30.661475 9030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:35:30.666908 9030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:38349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:45945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:30.667892 9030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:30.668860 9030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:30.678705 9036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:30.678700 9038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:30.678700 9035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:30.678747 9030 server_base.cc:1034] running on GCE node
I20260928 06:35:30.679373 9030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:30.679996 9030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:30.681241 9030 hybrid_clock.cc:648] HybridClock initialized: now 1790577330681216 us; error 35 us; skew 500 ppm
Sep 28 06:35:30 dist-test-slave-j0qf krb5kdc[8787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577330, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:30.684233 9030 init.cc:377] Logged in from keytab as kudu/127.5.22.2@KRBTEST.COM (short username kudu)
I20260928 06:35:30.685186 9030 webserver.cc:460] Webserver started at http://127.5.22.2:45621/ using document root <none> and password file <none>
I20260928 06:35:30.685704 9030 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:30.685762 9030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:30.685949 9030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:30.688099 9030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "cc10cff75dbc4d8ead07d06bd0668aaa"
format_stamp: "Formatted at 2026-09-28 06:35:30 on dist-test-slave-j0qf"
server_key: "4f33817ee753253514b4d1a630236d6e"
server_key_iv: "a2a700582ea0e9273ca45ffe8000f6c0"
server_key_version: "encryptionkey@0"
I20260928 06:35:30.688606 9030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "cc10cff75dbc4d8ead07d06bd0668aaa"
format_stamp: "Formatted at 2026-09-28 06:35:30 on dist-test-slave-j0qf"
server_key: "4f33817ee753253514b4d1a630236d6e"
server_key_iv: "a2a700582ea0e9273ca45ffe8000f6c0"
server_key_version: "encryptionkey@0"
I20260928 06:35:30.692662 9030 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 06:35:30.695066 9045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:30.696358 9030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:30.696493 9030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "cc10cff75dbc4d8ead07d06bd0668aaa"
format_stamp: "Formatted at 2026-09-28 06:35:30 on dist-test-slave-j0qf"
server_key: "4f33817ee753253514b4d1a630236d6e"
server_key_iv: "a2a700582ea0e9273ca45ffe8000f6c0"
server_key_version: "encryptionkey@0"
I20260928 06:35:30.696643 9030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:30.719384 9030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:30.722481 9030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:30.722700 9030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:30.723347 9030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:30.724225 9030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:30.724272 9030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:30.724340 9030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:30.724380 9030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:30.733254 9030 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:44589
I20260928 06:35:30.733278 9159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:44589 every 8 connection(s)
I20260928 06:35:30.734171 9030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:30.742141 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 9030
I20260928 06:35:30.742271 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:30.742592 5208 external_mini_cluster.cc:1428] Setting key 6519ab54cd790f1f3e9efb8c1a094744
Sep 28 06:35:30 dist-test-slave-j0qf krb5kdc[8787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577330, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:30.749075 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:30.736155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:59465 (local address 127.5.22.62:45945)
0928 06:35:30.736322 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:30.736326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:30.737172 (+ 846us) server_negotiation.cc:407] Connection header received
0928 06:35:30.738680 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:30.738683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:30.738745 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:30.738823 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:30.740103 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:30.740705 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:30.742080 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:30.742445 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:30.742534 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:30.745197 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:30.745200 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:30.745202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:30.745224 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:30.746759 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:30.747288 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:30.747293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:30.747295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:30.747380 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:30.747728 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:30.747733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:30.747735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:30.747921 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:30.748032 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:30.748736 (+ 704us) server_negotiation.cc:299] Negotiation successful
0928 06:35:30.748876 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 06:35:30.749956 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:30.736542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:45945 (local address 127.5.22.2:59465)
0928 06:35:30.736998 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:30.737047 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:30.738457 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:30.738990 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:30.738997 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:30.739352 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:30.739940 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:30.739956 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:30.740850 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:30.740855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:30.741905 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:30.741918 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:30.742612 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:30.742618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:30.742812 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:30.743441 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:30.743456 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:30.745015 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:30.746883 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:30.746889 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:30.746903 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:30.747180 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:30.747476 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:30.747479 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:30.747481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:30.747630 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:30.748030 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:30.748037 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:30.748380 (+ 343us) client_negotiation.cc:769] Sending connection context
0928 06:35:30.748671 (+ 291us) client_negotiation.cc:240] Negotiation successful
0928 06:35:30.749036 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":115,"threads_started":1}
I20260928 06:35:30.751336 9160 heartbeater.cc:347] Connected to a master server at 127.5.22.62:45945
I20260928 06:35:30.751600 9160 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.22.3@KRBTEST.COM; defaulting to no policy
I20260928 06:35:30.752137 9160 heartbeater.cc:511] Master 127.5.22.62:45945 requested a full tablet report, sending...
I20260928 06:35:30.753268 8832 ts_manager.cc:194] Registered new tserver with Master: cc10cff75dbc4d8ead07d06bd0668aaa (127.5.22.2:44589)
I20260928 06:35:30.753901 8832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.2@KRBTEST.COM'} at 127.5.22.2:59465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
I20260928 06:35:30.774121 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:45945
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:38349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:30.890641 9165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:30.890916 9165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:30.891001 9165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:30.895183 9165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:30.895252 9165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:30.895313 9165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:35:30.900839 9165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:38349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:45945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:30.901911 9165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:30.902990 9165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:30.912717 9173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:30.912742 9170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:30.912729 9171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:30.913115 9165 server_base.cc:1034] running on GCE node
I20260928 06:35:30.913480 9165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:30.914042 9165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:30.915195 9165 hybrid_clock.cc:648] HybridClock initialized: now 1790577330915154 us; error 57 us; skew 500 ppm
Sep 28 06:35:30 dist-test-slave-j0qf krb5kdc[8787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577330, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:30.917760 9165 init.cc:377] Logged in from keytab as kudu/127.5.22.3@KRBTEST.COM (short username kudu)
I20260928 06:35:30.918725 9165 webserver.cc:460] Webserver started at http://127.5.22.3:37255/ using document root <none> and password file <none>
I20260928 06:35:30.919281 9165 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:30.919369 9165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:30.919600 9165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:30.921648 9165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "aabeaa603c04459284df9fd38607a87b"
format_stamp: "Formatted at 2026-09-28 06:35:30 on dist-test-slave-j0qf"
server_key: "5e78ae460e4f0f124aa3ee5060025d99"
server_key_iv: "36777c3b0d317be58fffe97607e7fc9a"
server_key_version: "encryptionkey@0"
I20260928 06:35:30.922216 9165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "aabeaa603c04459284df9fd38607a87b"
format_stamp: "Formatted at 2026-09-28 06:35:30 on dist-test-slave-j0qf"
server_key: "5e78ae460e4f0f124aa3ee5060025d99"
server_key_iv: "36777c3b0d317be58fffe97607e7fc9a"
server_key_version: "encryptionkey@0"
I20260928 06:35:30.925884 9165 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:35:30.928452 9180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:30.929817 9165 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 06:35:30.929971 9165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "aabeaa603c04459284df9fd38607a87b"
format_stamp: "Formatted at 2026-09-28 06:35:30 on dist-test-slave-j0qf"
server_key: "5e78ae460e4f0f124aa3ee5060025d99"
server_key_iv: "36777c3b0d317be58fffe97607e7fc9a"
server_key_version: "encryptionkey@0"
I20260928 06:35:30.930101 9165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:30.952294 9165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:30.955391 9165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:30.955612 9165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:30.956228 9165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:30.957053 9165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:30.957129 9165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:30.957202 9165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:30.957238 9165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:30.966324 9165 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:38909
I20260928 06:35:30.966391 9293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:38909 every 8 connection(s)
I20260928 06:35:30.967675 9165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:30.971084 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 9165
I20260928 06:35:30.971225 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:30.971534 5208 external_mini_cluster.cc:1428] Setting key 7452846c246525386089c47a4a2877b3
Sep 28 06:35:30 dist-test-slave-j0qf krb5kdc[8787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577330, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:30.983503 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:30.970349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:38745 (local address 127.5.22.62:45945)
0928 06:35:30.970496 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:30.970500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:30.971730 (+ 1230us) server_negotiation.cc:407] Connection header received
0928 06:35:30.973688 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:30.973694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:30.973769 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:30.973854 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:30.975433 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:30.976143 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:30.977340 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:30.977581 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:30.977693 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:30.980163 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:30.980167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:30.980170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:30.980196 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:30.981413 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:30.981929 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:30.981932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:30.981934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:30.982020 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:30.982405 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:30.982409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:30.982411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:30.982601 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:30.982711 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:30.983209 (+ 498us) server_negotiation.cc:299] Negotiation successful
0928 06:35:30.983331 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 06:35:30.984282 9296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:30.970845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:45945 (local address 127.5.22.3:38745)
0928 06:35:30.971564 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:30.971626 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:30.973380 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:30.974071 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:30.974080 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:30.974634 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:30.975283 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:30.975300 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:30.976251 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:30.976255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:30.977210 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:30.977219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:30.977729 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:30.977733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:30.977912 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:30.978524 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:30.978540 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:30.979979 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:30.981533 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:30.981539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:30.981562 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:30.981815 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:30.982119 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:30.982124 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:30.982126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:30.982265 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:30.982719 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:30.982728 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:30.982971 (+ 243us) client_negotiation.cc:769] Sending connection context
0928 06:35:30.983152 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 06:35:30.983470 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":162,"threads_started":1}
I20260928 06:35:30.985526 9294 heartbeater.cc:347] Connected to a master server at 127.5.22.62:45945
I20260928 06:35:30.985778 9294 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:30.986455 9294 heartbeater.cc:511] Master 127.5.22.62:45945 requested a full tablet report, sending...
I20260928 06:35:30.987550 8832 ts_manager.cc:194] Registered new tserver with Master: aabeaa603c04459284df9fd38607a87b (127.5.22.3:38909)
I20260928 06:35:30.988109 8832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.3@KRBTEST.COM'} at 127.5.22.3:38745
I20260928 06:35:30.998243 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:35:31 dist-test-slave-j0qf krb5kdc[8787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:35:31 dist-test-slave-j0qf krb5kdc[8787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.1@KRBTEST.COM
I20260928 06:35:31.024932 9301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:31.015686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42210 (local address 127.5.22.1:38777)
0928 06:35:31.016046 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:31.016061 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:31.016085 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:35:31.016136 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:31.016141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:31.016324 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:31.016472 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:31.017128 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:31.017928 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:31.018819 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:31.019184 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:31.019268 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:31.021588 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:31.021596 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:31.021608 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:31.021632 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:31.023177 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:31.023599 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:31.023604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:31.023605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:31.023695 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:31.023989 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:31.023991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:31.023992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:31.024212 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:31.024374 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:31.024562 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 06:35:31.024719 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":161,"threads_started":1}
Sep 28 06:35:31 dist-test-slave-j0qf krb5kdc[8787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:31.036500 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:31.028040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37888 (local address 127.5.22.62:45945)
0928 06:35:31.028179 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:31.028185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:31.028266 (+ 81us) server_negotiation.cc:407] Connection header received
0928 06:35:31.028431 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:31.028434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:31.028477 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:31.028569 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:31.029232 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:31.029757 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:31.030605 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:31.030824 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:31.030930 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:31.033431 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:31.033434 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:31.033436 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:31.033459 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:31.034972 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:31.035340 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:31.035342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:31.035343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:31.035414 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:31.035652 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:31.035654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:31.035655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:31.035893 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:31.036019 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:31.036287 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 06:35:31.036378 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 06:35:31.036942 8832 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37888
I20260928 06:35:31.037685 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 8895
I20260928 06:35:31.044581 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 9030
I20260928 06:35:31.050022 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 9165
I20260928 06:35:31.055363 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 8800
2026-09-28T06:35:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2212 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:31 dist-test-slave-j0qf krb5kdc[9306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:31 dist-test-slave-j0qf krb5kdc[9306](info): set up 2 sockets
Sep 28 06:35:31 dist-test-slave-j0qf krb5kdc[9306](info): commencing operation
krb5kdc: starting...
W20260928 06:35:32.176663 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260928 06:35:32.350886 5208 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.174s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260928 06:35:32.519376 5208 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.102s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/test-user.keytab.
W20260928 06:35:33.304703 5208 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.785s user 0.000s sys 0.002s
Sep 28 06:35:33 dist-test-slave-j0qf krb5kdc[9306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
W20260928 06:35:33.501072 5208 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.5.22.62: real 0.173s user 0.000s sys 0.003s
I20260928 06:35:33.501628 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:41933
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:45079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:41933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:33.616241 9320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:33.616531 9320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:33.616575 9320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:33.620711 9320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:33.620781 9320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:33.620800 9320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:33.620818 9320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:33.620833 9320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:33.626121 9320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:45079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:41933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:41933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:33.627378 9320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:33.628434 9320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:33.637125 9325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:33.637131 9326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:33.637125 9328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:33.637399 9320 server_base.cc:1034] running on GCE node
I20260928 06:35:33.638278 9320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:33.639647 9320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:33.640829 9320 hybrid_clock.cc:648] HybridClock initialized: now 1790577333640807 us; error 36 us; skew 500 ppm
Sep 28 06:35:33 dist-test-slave-j0qf krb5kdc[9306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577333, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:33.648397 9320 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:35:33.649307 9320 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:33.649376 9320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:33.649603 9320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:33.651793 9320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "b78a2f6258c7492bbcaba190332197b4"
format_stamp: "Formatted at 2026-09-28 06:35:33 on dist-test-slave-j0qf"
server_key: "d16b00ec8585f1ef18be4318f6a2a623"
server_key_iv: "17df2637564901a235e4fd3444085912"
server_key_version: "encryptionkey@0"
I20260928 06:35:33.652390 9320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "b78a2f6258c7492bbcaba190332197b4"
format_stamp: "Formatted at 2026-09-28 06:35:33 on dist-test-slave-j0qf"
server_key: "d16b00ec8585f1ef18be4318f6a2a623"
server_key_iv: "17df2637564901a235e4fd3444085912"
server_key_version: "encryptionkey@0"
I20260928 06:35:33.656208 9320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:35:33.658928 9334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:33.660267 9320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:33.660422 9320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "b78a2f6258c7492bbcaba190332197b4"
format_stamp: "Formatted at 2026-09-28 06:35:33 on dist-test-slave-j0qf"
server_key: "d16b00ec8585f1ef18be4318f6a2a623"
server_key_iv: "17df2637564901a235e4fd3444085912"
server_key_version: "encryptionkey@0"
I20260928 06:35:33.660567 9320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:33.684608 9320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:33.688303 9320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:33.688506 9320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:33.695842 9320 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:41933
I20260928 06:35:33.696139 9386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:41933 every 8 connection(s)
I20260928 06:35:33.696587 9320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:33.698431 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 9320
I20260928 06:35:33.698571 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:35:33.698851 5208 external_mini_cluster.cc:1428] Setting key fb412ac6afafdbc532946932dc888c09
I20260928 06:35:33.699436 9387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:33.705443 9387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b78a2f6258c7492bbcaba190332197b4: Bootstrap starting.
Sep 28 06:35:33 dist-test-slave-j0qf krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:33.707773 9387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b78a2f6258c7492bbcaba190332197b4: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:33.708626 9387 log.cc:824] T 00000000000000000000000000000000 P b78a2f6258c7492bbcaba190332197b4: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:33.710696 9387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b78a2f6258c7492bbcaba190332197b4: No bootstrap required, opened a new log
I20260928 06:35:33.712661 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:33.700284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38214 (local address 127.5.22.62:41933)
0928 06:35:33.700763 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:33.700779 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:33.700817 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:35:33.701488 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:33.701505 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:33.701844 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:33.702184 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:33.703011 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:33.704006 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:33.704854 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:33.705326 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:33.705501 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:33.707909 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:33.707916 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:33.707936 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:33.707963 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:33.709780 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:33.710224 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:33.710229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:33.710233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:33.710411 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:33.710701 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:33.710704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:33.710706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:33.711147 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:33.711345 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:33.711647 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 06:35:33.711930 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":156,"threads_started":1}
I20260928 06:35:33.714535 9387 raft_consensus.cc:348] T 00000000000000000000000000000000 P b78a2f6258c7492bbcaba190332197b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78a2f6258c7492bbcaba190332197b4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 41933 } }
I20260928 06:35:33.714809 9387 raft_consensus.cc:374] T 00000000000000000000000000000000 P b78a2f6258c7492bbcaba190332197b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:33.714895 9387 raft_consensus.cc:729] T 00000000000000000000000000000000 P b78a2f6258c7492bbcaba190332197b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b78a2f6258c7492bbcaba190332197b4, State: Initialized, Role: FOLLOWER
I20260928 06:35:33.715310 9387 consensus_queue.cc:260] T 00000000000000000000000000000000 P b78a2f6258c7492bbcaba190332197b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78a2f6258c7492bbcaba190332197b4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 41933 } }
I20260928 06:35:33.715433 9387 raft_consensus.cc:388] T 00000000000000000000000000000000 P b78a2f6258c7492bbcaba190332197b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:33.715479 9387 raft_consensus.cc:482] T 00000000000000000000000000000000 P b78a2f6258c7492bbcaba190332197b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:33.715552 9387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b78a2f6258c7492bbcaba190332197b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:33.716390 9387 raft_consensus.cc:504] T 00000000000000000000000000000000 P b78a2f6258c7492bbcaba190332197b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78a2f6258c7492bbcaba190332197b4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 41933 } }
I20260928 06:35:33.716709 9387 leader_election.cc:302] T 00000000000000000000000000000000 P b78a2f6258c7492bbcaba190332197b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b78a2f6258c7492bbcaba190332197b4; no voters:
I20260928 06:35:33.716956 9387 leader_election.cc:288] T 00000000000000000000000000000000 P b78a2f6258c7492bbcaba190332197b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:33.717111 9392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b78a2f6258c7492bbcaba190332197b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:33.717370 9392 raft_consensus.cc:686] T 00000000000000000000000000000000 P b78a2f6258c7492bbcaba190332197b4 [term 1 LEADER]: Becoming Leader. State: Replica: b78a2f6258c7492bbcaba190332197b4, State: Running, Role: LEADER
I20260928 06:35:33.717669 9392 consensus_queue.cc:237] T 00000000000000000000000000000000 P b78a2f6258c7492bbcaba190332197b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78a2f6258c7492bbcaba190332197b4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 41933 } }
I20260928 06:35:33.718180 9387 sys_catalog.cc:564] T 00000000000000000000000000000000 P b78a2f6258c7492bbcaba190332197b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:33.719343 9393 sys_catalog.cc:455] T 00000000000000000000000000000000 P b78a2f6258c7492bbcaba190332197b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b78a2f6258c7492bbcaba190332197b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78a2f6258c7492bbcaba190332197b4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 41933 } } }
I20260928 06:35:33.719413 9394 sys_catalog.cc:455] T 00000000000000000000000000000000 P b78a2f6258c7492bbcaba190332197b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b78a2f6258c7492bbcaba190332197b4. Latest consensus state: current_term: 1 leader_uuid: "b78a2f6258c7492bbcaba190332197b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78a2f6258c7492bbcaba190332197b4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 41933 } } }
I20260928 06:35:33.719471 9393 sys_catalog.cc:458] T 00000000000000000000000000000000 P b78a2f6258c7492bbcaba190332197b4 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:33.719527 9394 sys_catalog.cc:458] T 00000000000000000000000000000000 P b78a2f6258c7492bbcaba190332197b4 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:33.719820 9401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:33.723178 9401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:33.728811 9401 catalog_manager.cc:1347] Generated new cluster ID: 6bacac74cf254f0099236531685b6691
I20260928 06:35:33.728896 9401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:33.753268 9401 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:33.754681 9401 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:33.776511 9401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b78a2f6258c7492bbcaba190332197b4: Generated new TSK 0
I20260928 06:35:33.777451 9401 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.22.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
I20260928 06:35:33.810086 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:41933
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:45079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:33.922525 9413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:33.922791 9413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:33.922837 9413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:33.926890 9413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:33.926949 9413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:33.927039 9413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:35:33.932344 9413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:45079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:41933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:33.933425 9413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:33.934494 9413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:33.944350 9418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:33.944391 9413 server_base.cc:1034] running on GCE node
W20260928 06:35:33.944489 9421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:33.944408 9419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:33.945039 9413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:33.945648 9413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:33.946828 9413 hybrid_clock.cc:648] HybridClock initialized: now 1790577333946813 us; error 30 us; skew 500 ppm
Sep 28 06:35:33 dist-test-slave-j0qf krb5kdc[9306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577333, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:33.949977 9413 init.cc:377] Logged in from keytab as kudu/127.5.22.1@KRBTEST.COM (short username kudu)
I20260928 06:35:33.950740 9413 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:33.950837 9413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:33.951088 9413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:33.953117 9413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "7385d5be9f0849e080b4f9b4b4759094"
format_stamp: "Formatted at 2026-09-28 06:35:33 on dist-test-slave-j0qf"
server_key: "79bd34a38d20745f7584a84d4eea2718"
server_key_iv: "5630838689fffb9cc72cc77671a96f0f"
server_key_version: "encryptionkey@0"
I20260928 06:35:33.953670 9413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "7385d5be9f0849e080b4f9b4b4759094"
format_stamp: "Formatted at 2026-09-28 06:35:33 on dist-test-slave-j0qf"
server_key: "79bd34a38d20745f7584a84d4eea2718"
server_key_iv: "5630838689fffb9cc72cc77671a96f0f"
server_key_version: "encryptionkey@0"
I20260928 06:35:33.957242 9413 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 06:35:33.959580 9427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:33.960903 9413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:35:33.961045 9413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "7385d5be9f0849e080b4f9b4b4759094"
format_stamp: "Formatted at 2026-09-28 06:35:33 on dist-test-slave-j0qf"
server_key: "79bd34a38d20745f7584a84d4eea2718"
server_key_iv: "5630838689fffb9cc72cc77671a96f0f"
server_key_version: "encryptionkey@0"
I20260928 06:35:33.961153 9413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:33.973423 9413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:33.976795 9413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:33.976994 9413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:33.977594 9413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:33.978457 9413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:33.978533 9413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:33.978602 9413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:33.978652 9413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:33.987109 9413 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:45927
I20260928 06:35:33.987130 9540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:45927 every 8 connection(s)
I20260928 06:35:33.987728 9413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:33.996541 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 9413
I20260928 06:35:33.996690 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:33.997022 5208 external_mini_cluster.cc:1428] Setting key 53971e89a70a5e755fae826764c00d32
Sep 28 06:35:33 dist-test-slave-j0qf krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577333, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:34.001956 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:33.989421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:43763 (local address 127.5.22.62:41933)
0928 06:35:33.989578 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:33.989584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:33.990466 (+ 882us) server_negotiation.cc:407] Connection header received
0928 06:35:33.991814 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:33.991817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:33.991876 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:33.991955 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:33.993118 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:33.993672 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:33.994983 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:33.995227 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:33.995337 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:33.997976 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:33.997980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:33.997987 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:33.998012 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:33.999522 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:34.000151 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:34.000154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:34.000156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:34.000240 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:34.000643 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:34.000645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:34.000647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:34.000823 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:34.000898 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:34.001557 (+ 659us) server_negotiation.cc:299] Negotiation successful
0928 06:35:34.001711 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 06:35:34.002740 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:33.989826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:41933 (local address 127.5.22.1:43763)
0928 06:35:33.990276 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:33.990362 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:33.991609 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:33.992126 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:33.992133 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:33.992489 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:33.992972 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:33.992985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:33.993800 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:33.993803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:33.994865 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:33.994874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:33.995376 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:33.995381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:33.995565 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:33.996152 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:33.996168 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:33.997811 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:33.999648 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:33.999654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:33.999670 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:34.000025 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:34.000330 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:34.000335 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:34.000338 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:34.000543 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:34.000929 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:34.000939 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:34.001259 (+ 320us) client_negotiation.cc:769] Sending connection context
0928 06:35:34.001466 (+ 207us) client_negotiation.cc:240] Negotiation successful
0928 06:35:34.001734 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":116,"threads_started":1}
I20260928 06:35:34.004040 9541 heartbeater.cc:347] Connected to a master server at 127.5.22.62:41933
I20260928 06:35:34.004277 9541 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:34.004721 9541 heartbeater.cc:511] Master 127.5.22.62:41933 requested a full tablet report, sending...
I20260928 06:35:34.006412 9351 ts_manager.cc:194] Registered new tserver with Master: 7385d5be9f0849e080b4f9b4b4759094 (127.5.22.1:45927)
WARNING: no policy specified for kudu/127.5.22.2@KRBTEST.COM; defaulting to no policy
I20260928 06:35:34.007702 9351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.1@KRBTEST.COM'} at 127.5.22.1:43763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
I20260928 06:35:34.025920 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:41933
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:45079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:34.136149 9546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:34.136404 9546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:34.136447 9546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:34.140453 9546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:34.140506 9546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:34.140563 9546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:35:34.145763 9546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:45079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:41933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:34.146720 9546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:34.147670 9546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:34.156984 9552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:34.157032 9551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:34.156978 9554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:34.157333 9546 server_base.cc:1034] running on GCE node
I20260928 06:35:34.157836 9546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:34.158547 9546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:34.159751 9546 hybrid_clock.cc:648] HybridClock initialized: now 1790577334159743 us; error 42 us; skew 500 ppm
Sep 28 06:35:34 dist-test-slave-j0qf krb5kdc[9306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577334, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:34.162690 9546 init.cc:377] Logged in from keytab as kudu/127.5.22.2@KRBTEST.COM (short username kudu)
I20260928 06:35:34.163440 9546 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:34.163506 9546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:34.163714 9546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:34.165912 9546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "a7fdf0305dfb4be29e90b0be55e1e6a3"
format_stamp: "Formatted at 2026-09-28 06:35:34 on dist-test-slave-j0qf"
server_key: "41662f097c088ead9d4f59f16f936f4f"
server_key_iv: "5c0fa85e6d281c4e5fa1998e95f56fe4"
server_key_version: "encryptionkey@0"
I20260928 06:35:34.166496 9546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "a7fdf0305dfb4be29e90b0be55e1e6a3"
format_stamp: "Formatted at 2026-09-28 06:35:34 on dist-test-slave-j0qf"
server_key: "41662f097c088ead9d4f59f16f936f4f"
server_key_iv: "5c0fa85e6d281c4e5fa1998e95f56fe4"
server_key_version: "encryptionkey@0"
I20260928 06:35:34.170207 9546 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:34.172945 9560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:34.174281 9546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:35:34.174484 9546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "a7fdf0305dfb4be29e90b0be55e1e6a3"
format_stamp: "Formatted at 2026-09-28 06:35:34 on dist-test-slave-j0qf"
server_key: "41662f097c088ead9d4f59f16f936f4f"
server_key_iv: "5c0fa85e6d281c4e5fa1998e95f56fe4"
server_key_version: "encryptionkey@0"
I20260928 06:35:34.174609 9546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:34.184576 9546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:34.187465 9546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:34.187669 9546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:34.188241 9546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:34.189075 9546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:34.189122 9546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:34.189188 9546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:34.189224 9546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:34.198447 9546 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:38859
I20260928 06:35:34.198493 9674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:38859 every 8 connection(s)
I20260928 06:35:34.199057 9546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:34.202203 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 9546
I20260928 06:35:34.202353 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:34.202633 5208 external_mini_cluster.cc:1428] Setting key 6b4c05235622a487b76573db45b94565
Sep 28 06:35:34 dist-test-slave-j0qf krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577334, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.22.3@KRBTEST.COM; defaulting to no policy
I20260928 06:35:34.214068 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:34.200957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:58443 (local address 127.5.22.62:41933)
0928 06:35:34.201134 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:34.201138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:34.201983 (+ 845us) server_negotiation.cc:407] Connection header received
0928 06:35:34.203510 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:34.203514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:34.203623 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:34.203716 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:34.204904 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.205617 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.206852 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.207112 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.207225 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:34.210048 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:34.210052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:34.210054 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:34.210077 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:34.211654 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:34.212249 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:34.212252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:34.212253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:34.212329 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:34.212875 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:34.212881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:34.212886 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:34.213115 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:34.213194 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:34.213821 (+ 627us) server_negotiation.cc:299] Negotiation successful
0928 06:35:34.213924 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.3@KRBTEST.COM" created.
I20260928 06:35:34.215031 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:34.201324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:41933 (local address 127.5.22.2:58443)
0928 06:35:34.201803 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:34.201847 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:34.203292 (+ 1445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:34.203899 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:34.203907 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:34.204276 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:34.204760 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:34.204774 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.205762 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.205766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:34.206726 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:34.206736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.207266 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.207269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:34.207429 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:34.208022 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:34.208038 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:34.209864 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:34.211803 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:34.211809 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:34.211825 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:34.212157 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:34.212580 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:34.212583 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:34.212585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:34.212734 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:34.213233 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:34.213238 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:34.213480 (+ 242us) client_negotiation.cc:769] Sending connection context
0928 06:35:34.213684 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 06:35:34.213952 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":113,"threads_started":1}
I20260928 06:35:34.216223 9675 heartbeater.cc:347] Connected to a master server at 127.5.22.62:41933
I20260928 06:35:34.216476 9675 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:34.216960 9675 heartbeater.cc:511] Master 127.5.22.62:41933 requested a full tablet report, sending...
I20260928 06:35:34.218022 9351 ts_manager.cc:194] Registered new tserver with Master: a7fdf0305dfb4be29e90b0be55e1e6a3 (127.5.22.2:38859)
I20260928 06:35:34.218956 9351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.2@KRBTEST.COM'} at 127.5.22.2:58443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
I20260928 06:35:34.232934 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:41933
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:45079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:34.343927 9680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:34.344205 9680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:34.344252 9680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:34.348197 9680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:34.348251 9680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:34.348308 9680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:35:34.353470 9680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:45079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:41933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:34.354470 9680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:34.355404 9680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:34.364676 9685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:34.364703 9688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:34.364728 9680 server_base.cc:1034] running on GCE node
W20260928 06:35:34.364676 9686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:34.365337 9680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:34.366031 9680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:34.367187 9680 hybrid_clock.cc:648] HybridClock initialized: now 1790577334367140 us; error 59 us; skew 500 ppm
Sep 28 06:35:34 dist-test-slave-j0qf krb5kdc[9306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577334, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:34.370039 9680 init.cc:377] Logged in from keytab as kudu/127.5.22.3@KRBTEST.COM (short username kudu)
I20260928 06:35:34.370817 9680 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:34.370904 9680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:34.371137 9680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:34.373154 9680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "7e5f9c8a4f924c7db4f3e662660c172c"
format_stamp: "Formatted at 2026-09-28 06:35:34 on dist-test-slave-j0qf"
server_key: "c6a5ce15119eb1a8fe3fa99757beb365"
server_key_iv: "94c8f19244f76922937397b66b479603"
server_key_version: "encryptionkey@0"
I20260928 06:35:34.373710 9680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "7e5f9c8a4f924c7db4f3e662660c172c"
format_stamp: "Formatted at 2026-09-28 06:35:34 on dist-test-slave-j0qf"
server_key: "c6a5ce15119eb1a8fe3fa99757beb365"
server_key_iv: "94c8f19244f76922937397b66b479603"
server_key_version: "encryptionkey@0"
I20260928 06:35:34.377343 9680 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 06:35:34.379793 9694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:34.381116 9680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:35:34.381242 9680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "7e5f9c8a4f924c7db4f3e662660c172c"
format_stamp: "Formatted at 2026-09-28 06:35:34 on dist-test-slave-j0qf"
server_key: "c6a5ce15119eb1a8fe3fa99757beb365"
server_key_iv: "94c8f19244f76922937397b66b479603"
server_key_version: "encryptionkey@0"
I20260928 06:35:34.381346 9680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:34.394850 9680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:34.397787 9680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:34.397982 9680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:34.398607 9680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:34.399411 9680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:34.399477 9680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:34.399545 9680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:34.399597 9680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:34.408355 9680 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:45331
I20260928 06:35:34.408392 9808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:45331 every 8 connection(s)
I20260928 06:35:34.408958 9680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:35:34 dist-test-slave-j0qf krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577334, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:34.419239 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 9680
I20260928 06:35:34.419375 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:34.419818 5208 external_mini_cluster.cc:1428] Setting key ec8fe43f3bb49b82d41583bd7d94994f
I20260928 06:35:34.423060 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:34.410863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:40087 (local address 127.5.22.62:41933)
0928 06:35:34.410995 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:34.410999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:34.411889 (+ 890us) server_negotiation.cc:407] Connection header received
0928 06:35:34.413370 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:34.413373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:34.413431 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:34.413536 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:34.414743 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.415274 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.416390 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.416612 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.416713 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:34.419271 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:34.419275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:34.419278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:34.419310 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:34.421139 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:34.421636 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:34.421641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:34.421643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:34.421732 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:34.422069 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:34.422072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:34.422073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:34.422254 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:34.422389 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:34.422849 (+ 460us) server_negotiation.cc:299] Negotiation successful
0928 06:35:34.422945 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:35:34.423895 9811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:34.411290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:41933 (local address 127.5.22.3:40087)
0928 06:35:34.411746 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:34.411788 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:34.413144 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:34.413714 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:34.413721 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:34.414071 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:34.414606 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:34.414620 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.415394 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.415403 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:34.416269 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:34.416277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.416765 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.416769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:34.416925 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:34.417470 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:34.417485 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:34.419100 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:34.421254 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:34.421260 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:34.421275 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:34.421532 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:34.421839 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:34.421842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:34.421844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:34.421987 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:34.422385 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:34.422389 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:34.422646 (+ 257us) client_negotiation.cc:769] Sending connection context
0928 06:35:34.422801 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 06:35:34.423052 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":113,"threads_started":1}
I20260928 06:35:34.425030 9809 heartbeater.cc:347] Connected to a master server at 127.5.22.62:41933
I20260928 06:35:34.425252 9809 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:34.425638 9809 heartbeater.cc:511] Master 127.5.22.62:41933 requested a full tablet report, sending...
I20260928 06:35:34.426596 9351 ts_manager.cc:194] Registered new tserver with Master: 7e5f9c8a4f924c7db4f3e662660c172c (127.5.22.3:45331)
I20260928 06:35:34.427163 9351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.3@KRBTEST.COM'} at 127.5.22.3:40087
I20260928 06:35:34.433560 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:35:34.442507 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:34.435578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.5.22.62:41933)
0928 06:35:34.435697 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:34.435700 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:34.435792 (+ 92us) server_negotiation.cc:407] Connection header received
0928 06:35:34.435891 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:34.435893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:34.435931 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:34.436016 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:34.436688 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.437200 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.438010 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.438225 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.438373 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:34.439621 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:34.439624 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:34.439626 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:34.439646 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:34.441103 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:34.441496 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:34.441498 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:34.441500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:34.441570 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:34.441785 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:34.441787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:34.441788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:34.441982 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:34.442107 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:34.442255 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 06:35:34.442395 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:35:34.445344 9351 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38230:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:35:34.448315 9351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:34.463097 9821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:34.456593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34662 (local address 127.5.22.3:45331)
0928 06:35:34.457860 (+ 1267us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:34.457984 (+ 124us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:34.458005 (+ 21us) server_negotiation.cc:407] Connection header received
0928 06:35:34.458069 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:34.458073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:34.458271 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:34.458462 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:34.459052 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.459867 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.461794 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.462407 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.462520 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:34.462658 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:34.462827 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 06:35:34.462927 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1167,"thread_start_us":118,"threads_started":1}
I20260928 06:35:34.463152 9822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:34.456667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:38859 (local address 127.0.0.1:59708)
0928 06:35:34.457755 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:34.457767 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:34.457854 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:34.458451 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:34.458459 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:34.458481 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:34.458626 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:34.458636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.460045 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.460048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:34.461516 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:34.461527 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.462721 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.462725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:34.462799 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:34.462816 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 06:35:34.462902 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:35:34.462979 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"mutex_wait_us":35,"thread_start_us":103,"threads_started":1}
I20260928 06:35:34.463178 9825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:34.457281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59708 (local address 127.5.22.2:38859)
0928 06:35:34.457745 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:34.457757 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:34.457971 (+ 214us) server_negotiation.cc:407] Connection header received
0928 06:35:34.458059 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:34.458064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:34.458271 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:34.458462 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:34.458758 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.459935 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.461632 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.462395 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.462507 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:34.462657 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:34.462949 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 06:35:34.463044 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":135,"threads_started":1}
I20260928 06:35:34.463472 9820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:34.456781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:45927 (local address 127.0.0.1:40880)
0928 06:35:34.457872 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:34.457887 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:34.457982 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:34.458640 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:34.458643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:34.458674 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:34.458898 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:34.458904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.460298 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.460301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:34.461400 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:34.461410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.463048 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.463051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:34.463141 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:34.463172 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 06:35:34.463254 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 06:35:34.463339 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"mutex_wait_us":149,"spinlock_wait_cycles":3968,"thread_start_us":91,"threads_started":1}
I20260928 06:35:34.463924 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:34.457141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40880 (local address 127.5.22.1:45927)
0928 06:35:34.457506 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:34.457518 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:34.458082 (+ 564us) server_negotiation.cc:407] Connection header received
0928 06:35:34.458165 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:34.458169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:34.458353 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:34.458502 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:34.459021 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.460148 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.461522 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.462410 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.462518 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:34.462657 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:34.463630 (+ 973us) server_negotiation.cc:299] Negotiation successful
0928 06:35:34.463754 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":148,"threads_started":1}
I20260928 06:35:34.464772 9823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:34.456469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:45331 (local address 127.0.0.1:34662)
0928 06:35:34.457359 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:34.457408 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:34.457535 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:34.458640 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:34.458643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:34.458673 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:34.458895 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:34.458903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.459999 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.460006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:34.461687 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:34.461694 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.462520 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.462524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:34.462629 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:34.462666 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 06:35:34.462799 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 06:35:34.462927 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":503,"spinlock_wait_cycles":5120,"thread_start_us":86,"threads_started":1}
I20260928 06:35:34.466488 9608 tablet_service.cc:1467] Processing CreateTablet for tablet 4bdd0749541843fba942a3a0ca14fe3c (DEFAULT_TABLE table=test-table [id=4b05ec7509544be694e74250bb50ca50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:34.466646 9475 tablet_service.cc:1467] Processing CreateTablet for tablet 4bdd0749541843fba942a3a0ca14fe3c (DEFAULT_TABLE table=test-table [id=4b05ec7509544be694e74250bb50ca50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:34.467665 9608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bdd0749541843fba942a3a0ca14fe3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:34.467669 9475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bdd0749541843fba942a3a0ca14fe3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:34.467437 9743 tablet_service.cc:1467] Processing CreateTablet for tablet 4bdd0749541843fba942a3a0ca14fe3c (DEFAULT_TABLE table=test-table [id=4b05ec7509544be694e74250bb50ca50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:34.468313 9743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bdd0749541843fba942a3a0ca14fe3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:34.473336 9826 tablet_bootstrap.cc:492] T 4bdd0749541843fba942a3a0ca14fe3c P 7385d5be9f0849e080b4f9b4b4759094: Bootstrap starting.
I20260928 06:35:34.475371 9828 tablet_bootstrap.cc:492] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c: Bootstrap starting.
I20260928 06:35:34.475370 9827 tablet_bootstrap.cc:492] T 4bdd0749541843fba942a3a0ca14fe3c P a7fdf0305dfb4be29e90b0be55e1e6a3: Bootstrap starting.
I20260928 06:35:34.475687 9826 tablet_bootstrap.cc:654] T 4bdd0749541843fba942a3a0ca14fe3c P 7385d5be9f0849e080b4f9b4b4759094: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:34.476795 9826 log.cc:824] T 4bdd0749541843fba942a3a0ca14fe3c P 7385d5be9f0849e080b4f9b4b4759094: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:34.477488 9828 tablet_bootstrap.cc:654] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:34.477488 9827 tablet_bootstrap.cc:654] T 4bdd0749541843fba942a3a0ca14fe3c P a7fdf0305dfb4be29e90b0be55e1e6a3: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:34.478753 9826 tablet_bootstrap.cc:492] T 4bdd0749541843fba942a3a0ca14fe3c P 7385d5be9f0849e080b4f9b4b4759094: No bootstrap required, opened a new log
I20260928 06:35:34.478830 9828 log.cc:824] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:34.478830 9827 log.cc:824] T 4bdd0749541843fba942a3a0ca14fe3c P a7fdf0305dfb4be29e90b0be55e1e6a3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:34.478955 9826 ts_tablet_manager.cc:1397] T 4bdd0749541843fba942a3a0ca14fe3c P 7385d5be9f0849e080b4f9b4b4759094: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 06:35:34.480728 9827 tablet_bootstrap.cc:492] T 4bdd0749541843fba942a3a0ca14fe3c P a7fdf0305dfb4be29e90b0be55e1e6a3: No bootstrap required, opened a new log
I20260928 06:35:34.480949 9827 ts_tablet_manager.cc:1397] T 4bdd0749541843fba942a3a0ca14fe3c P a7fdf0305dfb4be29e90b0be55e1e6a3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 06:35:34.482026 9826 raft_consensus.cc:348] T 4bdd0749541843fba942a3a0ca14fe3c P 7385d5be9f0849e080b4f9b4b4759094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5f9c8a4f924c7db4f3e662660c172c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 45331 } } peers { permanent_uuid: "a7fdf0305dfb4be29e90b0be55e1e6a3" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 38859 } } peers { permanent_uuid: "7385d5be9f0849e080b4f9b4b4759094" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 45927 } }
I20260928 06:35:34.482275 9826 raft_consensus.cc:374] T 4bdd0749541843fba942a3a0ca14fe3c P 7385d5be9f0849e080b4f9b4b4759094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:34.482355 9826 raft_consensus.cc:729] T 4bdd0749541843fba942a3a0ca14fe3c P 7385d5be9f0849e080b4f9b4b4759094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7385d5be9f0849e080b4f9b4b4759094, State: Initialized, Role: FOLLOWER
I20260928 06:35:34.482795 9826 consensus_queue.cc:260] T 4bdd0749541843fba942a3a0ca14fe3c P 7385d5be9f0849e080b4f9b4b4759094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5f9c8a4f924c7db4f3e662660c172c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 45331 } } peers { permanent_uuid: "a7fdf0305dfb4be29e90b0be55e1e6a3" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 38859 } } peers { permanent_uuid: "7385d5be9f0849e080b4f9b4b4759094" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 45927 } }
I20260928 06:35:34.483479 9541 heartbeater.cc:503] Master 127.5.22.62:41933 was elected leader, sending a full tablet report...
I20260928 06:35:34.483760 9826 ts_tablet_manager.cc:1428] T 4bdd0749541843fba942a3a0ca14fe3c P 7385d5be9f0849e080b4f9b4b4759094: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260928 06:35:34.483916 9827 raft_consensus.cc:348] T 4bdd0749541843fba942a3a0ca14fe3c P a7fdf0305dfb4be29e90b0be55e1e6a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5f9c8a4f924c7db4f3e662660c172c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 45331 } } peers { permanent_uuid: "a7fdf0305dfb4be29e90b0be55e1e6a3" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 38859 } } peers { permanent_uuid: "7385d5be9f0849e080b4f9b4b4759094" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 45927 } }
I20260928 06:35:34.484163 9827 raft_consensus.cc:374] T 4bdd0749541843fba942a3a0ca14fe3c P a7fdf0305dfb4be29e90b0be55e1e6a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:34.484217 9827 raft_consensus.cc:729] T 4bdd0749541843fba942a3a0ca14fe3c P a7fdf0305dfb4be29e90b0be55e1e6a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7fdf0305dfb4be29e90b0be55e1e6a3, State: Initialized, Role: FOLLOWER
I20260928 06:35:34.484642 9827 consensus_queue.cc:260] T 4bdd0749541843fba942a3a0ca14fe3c P a7fdf0305dfb4be29e90b0be55e1e6a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5f9c8a4f924c7db4f3e662660c172c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 45331 } } peers { permanent_uuid: "a7fdf0305dfb4be29e90b0be55e1e6a3" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 38859 } } peers { permanent_uuid: "7385d5be9f0849e080b4f9b4b4759094" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 45927 } }
I20260928 06:35:34.485600 9827 ts_tablet_manager.cc:1428] T 4bdd0749541843fba942a3a0ca14fe3c P a7fdf0305dfb4be29e90b0be55e1e6a3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 06:35:34.485716 9828 tablet_bootstrap.cc:492] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c: No bootstrap required, opened a new log
I20260928 06:35:34.485910 9828 ts_tablet_manager.cc:1397] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260928 06:35:34.485951 9675 heartbeater.cc:503] Master 127.5.22.62:41933 was elected leader, sending a full tablet report...
W20260928 06:35:34.488667 9542 tablet.cc:2369] T 4bdd0749541843fba942a3a0ca14fe3c P 7385d5be9f0849e080b4f9b4b4759094: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:35:34.489435 9828 raft_consensus.cc:348] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5f9c8a4f924c7db4f3e662660c172c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 45331 } } peers { permanent_uuid: "a7fdf0305dfb4be29e90b0be55e1e6a3" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 38859 } } peers { permanent_uuid: "7385d5be9f0849e080b4f9b4b4759094" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 45927 } }
I20260928 06:35:34.489694 9828 raft_consensus.cc:374] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:34.489766 9828 raft_consensus.cc:729] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e5f9c8a4f924c7db4f3e662660c172c, State: Initialized, Role: FOLLOWER
I20260928 06:35:34.490157 9828 consensus_queue.cc:260] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5f9c8a4f924c7db4f3e662660c172c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 45331 } } peers { permanent_uuid: "a7fdf0305dfb4be29e90b0be55e1e6a3" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 38859 } } peers { permanent_uuid: "7385d5be9f0849e080b4f9b4b4759094" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 45927 } }
I20260928 06:35:34.490767 9809 heartbeater.cc:503] Master 127.5.22.62:41933 was elected leader, sending a full tablet report...
I20260928 06:35:34.491034 9828 ts_tablet_manager.cc:1428] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260928 06:35:34.659861 9810 tablet.cc:2369] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:35:34.688755 9834 raft_consensus.cc:482] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:34.688984 9834 raft_consensus.cc:504] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5f9c8a4f924c7db4f3e662660c172c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 45331 } } peers { permanent_uuid: "a7fdf0305dfb4be29e90b0be55e1e6a3" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 38859 } } peers { permanent_uuid: "7385d5be9f0849e080b4f9b4b4759094" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 45927 } }
I20260928 06:35:34.690204 9834 leader_election.cc:288] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7fdf0305dfb4be29e90b0be55e1e6a3 (127.5.22.2:38859), 7385d5be9f0849e080b4f9b4b4759094 (127.5.22.1:45927)
I20260928 06:35:34.694034 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:34.690614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:41469 (local address 127.5.22.1:45927)
0928 06:35:34.690746 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:34.690750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:34.690944 (+ 194us) server_negotiation.cc:407] Connection header received
0928 06:35:34.691004 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:34.691010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:34.691058 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:34.691134 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:34.691401 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.691910 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.693058 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.693444 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.693569 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:34.693631 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:34.693842 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 06:35:34.693908 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 06:35:34.694041 9825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:34.690514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:48215 (local address 127.5.22.2:38859)
0928 06:35:34.690649 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:34.690654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:34.690798 (+ 144us) server_negotiation.cc:407] Connection header received
0928 06:35:34.690864 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:34.690866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:34.690935 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:34.691021 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:34.691423 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.691907 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.693062 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.693447 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.693557 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:34.693621 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:34.693838 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 06:35:34.693905 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 06:35:34.694036 9811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:34.690440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:38859 (local address 127.5.22.3:48215)
0928 06:35:34.690629 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:34.690644 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:34.690757 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:34.691046 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:34.691048 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:34.691060 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:34.691269 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:34.691278 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.692016 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.692022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:34.692930 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:34.692940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.693562 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.693565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:34.693706 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:34.693732 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 06:35:34.693812 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 06:35:34.693885 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":76}
I20260928 06:35:34.694054 9835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:34.690507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:45927 (local address 127.5.22.3:41469)
0928 06:35:34.690848 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:34.690867 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:34.690937 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:34.691149 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:34.691151 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:34.691166 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:34.691297 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:34.691302 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.692041 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.692044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:34.692931 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:34.692941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.693565 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.693568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:34.693703 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:34.693730 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 06:35:34.693818 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:35:34.693888 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":79,"thread_start_us":128,"threads_started":1}
I20260928 06:35:34.694782 9495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bdd0749541843fba942a3a0ca14fe3c" candidate_uuid: "7e5f9c8a4f924c7db4f3e662660c172c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7385d5be9f0849e080b4f9b4b4759094" is_pre_election: true
I20260928 06:35:34.694783 9629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bdd0749541843fba942a3a0ca14fe3c" candidate_uuid: "7e5f9c8a4f924c7db4f3e662660c172c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7fdf0305dfb4be29e90b0be55e1e6a3" is_pre_election: true
I20260928 06:35:34.695092 9629 raft_consensus.cc:2454] T 4bdd0749541843fba942a3a0ca14fe3c P a7fdf0305dfb4be29e90b0be55e1e6a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e5f9c8a4f924c7db4f3e662660c172c in term 0.
I20260928 06:35:34.695092 9495 raft_consensus.cc:2454] T 4bdd0749541843fba942a3a0ca14fe3c P 7385d5be9f0849e080b4f9b4b4759094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e5f9c8a4f924c7db4f3e662660c172c in term 0.
I20260928 06:35:34.695575 9695 leader_election.cc:302] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7385d5be9f0849e080b4f9b4b4759094, 7e5f9c8a4f924c7db4f3e662660c172c; no voters:
I20260928 06:35:34.695869 9834 raft_consensus.cc:2781] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:34.695950 9834 raft_consensus.cc:482] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:34.695989 9834 raft_consensus.cc:3037] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:34.697234 9834 raft_consensus.cc:504] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5f9c8a4f924c7db4f3e662660c172c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 45331 } } peers { permanent_uuid: "a7fdf0305dfb4be29e90b0be55e1e6a3" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 38859 } } peers { permanent_uuid: "7385d5be9f0849e080b4f9b4b4759094" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 45927 } }
I20260928 06:35:34.697587 9834 leader_election.cc:288] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c [CANDIDATE]: Term 1 election: Requested vote from peers a7fdf0305dfb4be29e90b0be55e1e6a3 (127.5.22.2:38859), 7385d5be9f0849e080b4f9b4b4759094 (127.5.22.1:45927)
I20260928 06:35:34.697921 9629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bdd0749541843fba942a3a0ca14fe3c" candidate_uuid: "7e5f9c8a4f924c7db4f3e662660c172c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7fdf0305dfb4be29e90b0be55e1e6a3"
I20260928 06:35:34.698028 9495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bdd0749541843fba942a3a0ca14fe3c" candidate_uuid: "7e5f9c8a4f924c7db4f3e662660c172c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7385d5be9f0849e080b4f9b4b4759094"
I20260928 06:35:34.698057 9629 raft_consensus.cc:3037] T 4bdd0749541843fba942a3a0ca14fe3c P a7fdf0305dfb4be29e90b0be55e1e6a3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:34.698154 9495 raft_consensus.cc:3037] T 4bdd0749541843fba942a3a0ca14fe3c P 7385d5be9f0849e080b4f9b4b4759094 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:34.699169 9629 raft_consensus.cc:2454] T 4bdd0749541843fba942a3a0ca14fe3c P a7fdf0305dfb4be29e90b0be55e1e6a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5f9c8a4f924c7db4f3e662660c172c in term 1.
I20260928 06:35:34.699357 9495 raft_consensus.cc:2454] T 4bdd0749541843fba942a3a0ca14fe3c P 7385d5be9f0849e080b4f9b4b4759094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5f9c8a4f924c7db4f3e662660c172c in term 1.
I20260928 06:35:34.699481 9697 leader_election.cc:302] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e5f9c8a4f924c7db4f3e662660c172c, a7fdf0305dfb4be29e90b0be55e1e6a3; no voters:
I20260928 06:35:34.699677 9834 raft_consensus.cc:2781] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:34.699920 9834 raft_consensus.cc:686] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c [term 1 LEADER]: Becoming Leader. State: Replica: 7e5f9c8a4f924c7db4f3e662660c172c, State: Running, Role: LEADER
W20260928 06:35:34.700089 9676 tablet.cc:2369] T 4bdd0749541843fba942a3a0ca14fe3c P a7fdf0305dfb4be29e90b0be55e1e6a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:35:34.700255 9834 consensus_queue.cc:237] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5f9c8a4f924c7db4f3e662660c172c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 45331 } } peers { permanent_uuid: "a7fdf0305dfb4be29e90b0be55e1e6a3" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 38859 } } peers { permanent_uuid: "7385d5be9f0849e080b4f9b4b4759094" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 45927 } }
I20260928 06:35:34.703346 9350 catalog_manager.cc:5495] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e5f9c8a4f924c7db4f3e662660c172c (127.5.22.3). New cstate: current_term: 1 leader_uuid: "7e5f9c8a4f924c7db4f3e662660c172c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5f9c8a4f924c7db4f3e662660c172c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 45331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7fdf0305dfb4be29e90b0be55e1e6a3" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7385d5be9f0849e080b4f9b4b4759094" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 45927 } health_report { overall_health: UNKNOWN } } }
I20260928 06:35:34.761252 9821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:34.757238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34676 (local address 127.5.22.3:45331)
0928 06:35:34.757393 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:34.757398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:34.757415 (+ 17us) server_negotiation.cc:407] Connection header received
0928 06:35:34.757595 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:34.757598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:34.757658 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:34.757781 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:34.758044 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.758629 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.759643 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:34.759891 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:34.760019 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:34.760342 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:34.760787 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:34.760936 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:34.761003 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 06:35:34.761067 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 06:35:34.767513 9629 raft_consensus.cc:1261] T 4bdd0749541843fba942a3a0ca14fe3c P a7fdf0305dfb4be29e90b0be55e1e6a3 [term 1 FOLLOWER]: Refusing update from remote peer 7e5f9c8a4f924c7db4f3e662660c172c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:34.767513 9495 raft_consensus.cc:1261] T 4bdd0749541843fba942a3a0ca14fe3c P 7385d5be9f0849e080b4f9b4b4759094 [term 1 FOLLOWER]: Refusing update from remote peer 7e5f9c8a4f924c7db4f3e662660c172c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:34.768237 9834 consensus_queue.cc:1035] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7385d5be9f0849e080b4f9b4b4759094" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 45927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:34.768445 9836 consensus_queue.cc:1035] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7fdf0305dfb4be29e90b0be55e1e6a3" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:34.775177 9839 mvcc.cc:205] Tried to move back new op lower bound from 7334204763200077824 to 7334204762934546432. Current Snapshot: MvccSnapshot[applied={T|T < 7334204763200077824}]
I20260928 06:35:34.778447 9840 mvcc.cc:205] Tried to move back new op lower bound from 7334204763200077824 to 7334204762934546432. Current Snapshot: MvccSnapshot[applied={T|T < 7334204763200077824}]
I20260928 06:35:34.785305 9350 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:34.785480 9350 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:34.787889 9350 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b78a2f6258c7492bbcaba190332197b4: Sending DeleteTablet for 3 replicas of tablet 4bdd0749541843fba942a3a0ca14fe3c
I20260928 06:35:34.788604 9743 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bdd0749541843fba942a3a0ca14fe3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:34 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:34662
I20260928 06:35:34.788643 9475 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bdd0749541843fba942a3a0ca14fe3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:34 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:40880
I20260928 06:35:34.788673 9608 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bdd0749541843fba942a3a0ca14fe3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:34 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:59708
I20260928 06:35:34.789250 9851 tablet_replica.cc:330] T 4bdd0749541843fba942a3a0ca14fe3c P a7fdf0305dfb4be29e90b0be55e1e6a3: stopping tablet replica
I20260928 06:35:34.789253 9850 tablet_replica.cc:330] T 4bdd0749541843fba942a3a0ca14fe3c P 7385d5be9f0849e080b4f9b4b4759094: stopping tablet replica
I20260928 06:35:34.789585 9850 raft_consensus.cc:2229] T 4bdd0749541843fba942a3a0ca14fe3c P 7385d5be9f0849e080b4f9b4b4759094 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:34.789584 9851 raft_consensus.cc:2229] T 4bdd0749541843fba942a3a0ca14fe3c P a7fdf0305dfb4be29e90b0be55e1e6a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:34.789906 9851 raft_consensus.cc:2258] T 4bdd0749541843fba942a3a0ca14fe3c P a7fdf0305dfb4be29e90b0be55e1e6a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:34.789907 9850 raft_consensus.cc:2258] T 4bdd0749541843fba942a3a0ca14fe3c P 7385d5be9f0849e080b4f9b4b4759094 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:34.790074 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 9413
I20260928 06:35:34.790261 9849 tablet_replica.cc:330] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c: stopping tablet replica
I20260928 06:35:34.790555 9849 raft_consensus.cc:2229] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:35:34.790946 9849 raft_consensus.cc:2258] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:34.792704 9849 ts_tablet_manager.cc:1905] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:34.792984 9851 ts_tablet_manager.cc:1905] T 4bdd0749541843fba942a3a0ca14fe3c P a7fdf0305dfb4be29e90b0be55e1e6a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:34.795619 9851 ts_tablet_manager.cc:1918] T 4bdd0749541843fba942a3a0ca14fe3c P a7fdf0305dfb4be29e90b0be55e1e6a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:34.795619 9849 ts_tablet_manager.cc:1918] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:34.795724 9851 log.cc:1196] T 4bdd0749541843fba942a3a0ca14fe3c P a7fdf0305dfb4be29e90b0be55e1e6a3: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-1/wal/wals/4bdd0749541843fba942a3a0ca14fe3c
I20260928 06:35:34.795845 9849 log.cc:1196] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/4bdd0749541843fba942a3a0ca14fe3c
I20260928 06:35:34.796063 9851 ts_tablet_manager.cc:1939] T 4bdd0749541843fba942a3a0ca14fe3c P a7fdf0305dfb4be29e90b0be55e1e6a3: Deleting consensus metadata
I20260928 06:35:34.796536 9849 ts_tablet_manager.cc:1939] T 4bdd0749541843fba942a3a0ca14fe3c P 7e5f9c8a4f924c7db4f3e662660c172c: Deleting consensus metadata
I20260928 06:35:34.797382 9337 catalog_manager.cc:4841] TS a7fdf0305dfb4be29e90b0be55e1e6a3 (127.5.22.2:38859): tablet 4bdd0749541843fba942a3a0ca14fe3c (table test-table [id=4b05ec7509544be694e74250bb50ca50]) successfully deleted
I20260928 06:35:34.797591 9338 catalog_manager.cc:4841] TS 7e5f9c8a4f924c7db4f3e662660c172c (127.5.22.3:45331): tablet 4bdd0749541843fba942a3a0ca14fe3c (table test-table [id=4b05ec7509544be694e74250bb50ca50]) successfully deleted
I20260928 06:35:34.800318 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 9546
W20260928 06:35:34.800329 9335 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.22.1:45927 (error 108)
I20260928 06:35:34.801862 9823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:34.801504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:45927 (local address 127.0.0.1:40892)
0928 06:35:34.801664 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:34.801754 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.1:45927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 06:35:34.802026 9335 catalog_manager.cc:4584] TS 7385d5be9f0849e080b4f9b4b4759094 (127.5.22.1:45927): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4bdd0749541843fba942a3a0ca14fe3c: Network error: Client connection negotiation failed: client connection to 127.5.22.1:45927: connect: Connection refused (error 111)
I20260928 06:35:34.807233 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 9680
I20260928 06:35:34.812991 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 9320
2026-09-28T06:35:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3757 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T06:35:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:34Z Disabled control of system clock
I20260928 06:35:34.844094 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:37371
--webserver_interface=127.5.22.62
--webserver_port=0
--builtin_ntp_servers=127.5.22.20:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:37371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:35:34.959885 9856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:34.960141 9856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:34.960177 9856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:34.964221 9856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:34.964273 9856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:34.964288 9856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:34.964329 9856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:34.964358 9856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:34.970860 9856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:37371
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:37371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:34.972469 9856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:34.974097 9856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:34.982764 9862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:34.982781 9863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:34.982947 9865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:34.983421 9856 server_base.cc:1034] running on GCE node
I20260928 06:35:34.984004 9856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:34.985586 9856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:34.986775 9856 hybrid_clock.cc:648] HybridClock initialized: now 1790577334986747 us; error 46 us; skew 500 ppm
I20260928 06:35:34.988686 9856 webserver.cc:460] Webserver started at http://127.5.22.62:35849/ using document root <none> and password file <none>
I20260928 06:35:34.989253 9856 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:34.989316 9856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:34.989611 9856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:34.991725 9856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "0690e7b46051471788e2887e1220aa4b"
format_stamp: "Formatted at 2026-09-28 06:35:34 on dist-test-slave-j0qf"
server_key: "61df2543812516ddaec0a787d3c7cc19"
server_key_iv: "a109489870bb3d06f629a059a7de9234"
server_key_version: "encryptionkey@0"
I20260928 06:35:34.992287 9856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "0690e7b46051471788e2887e1220aa4b"
format_stamp: "Formatted at 2026-09-28 06:35:34 on dist-test-slave-j0qf"
server_key: "61df2543812516ddaec0a787d3c7cc19"
server_key_iv: "a109489870bb3d06f629a059a7de9234"
server_key_version: "encryptionkey@0"
I20260928 06:35:34.996003 9856 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260928 06:35:34.998785 9871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:35.000085 9856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 06:35:35.000211 9856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "0690e7b46051471788e2887e1220aa4b"
format_stamp: "Formatted at 2026-09-28 06:35:34 on dist-test-slave-j0qf"
server_key: "61df2543812516ddaec0a787d3c7cc19"
server_key_iv: "a109489870bb3d06f629a059a7de9234"
server_key_version: "encryptionkey@0"
I20260928 06:35:35.000321 9856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:35.004992 9856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:35.005600 9856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:35.005788 9856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:35.012861 9856 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:37371
I20260928 06:35:35.012868 9924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:37371 every 8 connection(s)
I20260928 06:35:35.013772 9856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:35.016862 9925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:35.020167 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 9856
I20260928 06:35:35.020272 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:35:35.020556 5208 external_mini_cluster.cc:1428] Setting key 4bf50f69ab0f3cf784ea8dadf9ede633
I20260928 06:35:35.022766 9925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0690e7b46051471788e2887e1220aa4b: Bootstrap starting.
I20260928 06:35:35.025175 9925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0690e7b46051471788e2887e1220aa4b: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:35.025934 9925 log.cc:824] T 00000000000000000000000000000000 P 0690e7b46051471788e2887e1220aa4b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:35.026154 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.021961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33404 (local address 127.5.22.62:37371)
0928 06:35:35.022615 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:35.022631 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:35.022666 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:35:35.023334 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:35.023352 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:35.023741 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:35.024022 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:35.024843 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:35.024848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:35.024860 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:35.024870 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:35.024923 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:35.024939 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:35.025025 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:35.025177 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 06:35:35.025417 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":136,"threads_started":1}
I20260928 06:35:35.028014 9925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0690e7b46051471788e2887e1220aa4b: No bootstrap required, opened a new log
I20260928 06:35:35.031471 9925 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0690e7b46051471788e2887e1220aa4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0690e7b46051471788e2887e1220aa4b" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 37371 } }
I20260928 06:35:35.031682 9925 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0690e7b46051471788e2887e1220aa4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:35.031716 9925 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0690e7b46051471788e2887e1220aa4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0690e7b46051471788e2887e1220aa4b, State: Initialized, Role: FOLLOWER
I20260928 06:35:35.032047 9925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0690e7b46051471788e2887e1220aa4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0690e7b46051471788e2887e1220aa4b" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 37371 } }
I20260928 06:35:35.032153 9925 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0690e7b46051471788e2887e1220aa4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:35.032196 9925 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0690e7b46051471788e2887e1220aa4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:35.032262 9925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0690e7b46051471788e2887e1220aa4b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:35.033097 9925 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0690e7b46051471788e2887e1220aa4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0690e7b46051471788e2887e1220aa4b" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 37371 } }
I20260928 06:35:35.033370 9925 leader_election.cc:302] T 00000000000000000000000000000000 P 0690e7b46051471788e2887e1220aa4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0690e7b46051471788e2887e1220aa4b; no voters:
I20260928 06:35:35.033726 9925 leader_election.cc:288] T 00000000000000000000000000000000 P 0690e7b46051471788e2887e1220aa4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:35.033818 9930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0690e7b46051471788e2887e1220aa4b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:35.034054 9930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0690e7b46051471788e2887e1220aa4b [term 1 LEADER]: Becoming Leader. State: Replica: 0690e7b46051471788e2887e1220aa4b, State: Running, Role: LEADER
I20260928 06:35:35.034356 9930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0690e7b46051471788e2887e1220aa4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0690e7b46051471788e2887e1220aa4b" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 37371 } }
I20260928 06:35:35.034924 9925 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0690e7b46051471788e2887e1220aa4b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:35.035918 9931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0690e7b46051471788e2887e1220aa4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0690e7b46051471788e2887e1220aa4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0690e7b46051471788e2887e1220aa4b" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 37371 } } }
I20260928 06:35:35.036074 9931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0690e7b46051471788e2887e1220aa4b [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:35.036340 9932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0690e7b46051471788e2887e1220aa4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0690e7b46051471788e2887e1220aa4b. Latest consensus state: current_term: 1 leader_uuid: "0690e7b46051471788e2887e1220aa4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0690e7b46051471788e2887e1220aa4b" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 37371 } } }
I20260928 06:35:35.036430 9932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0690e7b46051471788e2887e1220aa4b [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:35.036547 9939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:35.039443 9939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:35.044725 9939 catalog_manager.cc:1347] Generated new cluster ID: 6d47d110b7924440b291fb3f4fda745b
I20260928 06:35:35.044788 9939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:35.058964 9939 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:35.059321 9939 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:35.067175 9939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0690e7b46051471788e2887e1220aa4b: Generated new TSK 0
I20260928 06:35:35.068082 9939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:35.071309 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:37371
--builtin_ntp_servers=127.5.22.20:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:35:35.197695 9949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:35.197952 9949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:35.198010 9949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:35.202001 9949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:35.202059 9949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:35.202142 9949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:35:35.207818 9949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:37371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:35.208971 9949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:35.210500 9949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:35.220451 9957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:35.220453 9955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:35.220453 9954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:35.220491 9949 server_base.cc:1034] running on GCE node
I20260928 06:35:35.221103 9949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:35.221710 9949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:35.222892 9949 hybrid_clock.cc:648] HybridClock initialized: now 1790577335222875 us; error 28 us; skew 500 ppm
I20260928 06:35:35.224838 9949 webserver.cc:460] Webserver started at http://127.5.22.1:45707/ using document root <none> and password file <none>
I20260928 06:35:35.225471 9949 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:35.225569 9949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:35.225817 9949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:35.228121 9949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "a83e94d3684f417a836302fd8d5569a2"
format_stamp: "Formatted at 2026-09-28 06:35:35 on dist-test-slave-j0qf"
server_key: "a44b568bc917b5e03cef8207f00097fa"
server_key_iv: "f950e7502bb2ae235f18bc8a37405c63"
server_key_version: "encryptionkey@0"
I20260928 06:35:35.228757 9949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "a83e94d3684f417a836302fd8d5569a2"
format_stamp: "Formatted at 2026-09-28 06:35:35 on dist-test-slave-j0qf"
server_key: "a44b568bc917b5e03cef8207f00097fa"
server_key_iv: "f950e7502bb2ae235f18bc8a37405c63"
server_key_version: "encryptionkey@0"
I20260928 06:35:35.232704 9949 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260928 06:35:35.235180 9963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:35.236730 9949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:35.236862 9949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "a83e94d3684f417a836302fd8d5569a2"
format_stamp: "Formatted at 2026-09-28 06:35:35 on dist-test-slave-j0qf"
server_key: "a44b568bc917b5e03cef8207f00097fa"
server_key_iv: "f950e7502bb2ae235f18bc8a37405c63"
server_key_version: "encryptionkey@0"
I20260928 06:35:35.236974 9949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:35.241624 9949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:35.242257 9949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:35.242494 9949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:35.243052 9949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:35.243942 9949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:35.244009 9949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:35.244072 9949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:35.244119 9949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:35.252720 9949 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:34571
I20260928 06:35:35.252753 10077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:34571 every 8 connection(s)
I20260928 06:35:35.253593 9949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:35.258499 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 9949
I20260928 06:35:35.258632 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:35.258960 5208 external_mini_cluster.cc:1428] Setting key 8e617ca1e33d9fca16c5a82dda2abdd0
I20260928 06:35:35.259768 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.255366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:38693 (local address 127.5.22.62:37371)
0928 06:35:35.255497 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:35.255501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:35.256600 (+ 1099us) server_negotiation.cc:407] Connection header received
0928 06:35:35.257922 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:35.257925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:35.257970 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:35.258082 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:35.259284 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:35.259288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:35.259291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:35.259304 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:35.259326 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:35.259340 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:35.259445 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:35.259586 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 06:35:35.259632 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:35:35.260659 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.255805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:37371 (local address 127.5.22.1:38693)
0928 06:35:35.256450 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:35.256489 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:35.257708 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:35.258252 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:35.258258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:35.258362 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:35.258990 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:35:35.258992 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:35.259068 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:35:35.259142 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:35.259428 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:35.259435 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:35.259441 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:35:35.259660 (+ 219us) client_negotiation.cc:240] Negotiation successful
0928 06:35:35.259973 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":116,"threads_started":1}
I20260928 06:35:35.260933 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:37371
--builtin_ntp_servers=127.5.22.20:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 06:35:35.261806 10078 heartbeater.cc:347] Connected to a master server at 127.5.22.62:37371
I20260928 06:35:35.262046 10078 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:35.262537 10078 heartbeater.cc:511] Master 127.5.22.62:37371 requested a full tablet report, sending...
I20260928 06:35:35.264123 9889 ts_manager.cc:194] Registered new tserver with Master: a83e94d3684f417a836302fd8d5569a2 (127.5.22.1:34571)
W20260928 06:35:35.371824 10081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:35.372085 10081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:35.372123 10081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:35.376089 10081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:35.376149 10081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:35.376204 10081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:35:35.380980 10081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:37371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:35.381992 10081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:35.383350 10081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:35.392905 10087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:35.392877 10089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:35.393098 10081 server_base.cc:1034] running on GCE node
W20260928 06:35:35.393240 10086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:35.393741 10081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:35.394430 10081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:35.395625 10081 hybrid_clock.cc:648] HybridClock initialized: now 1790577335395596 us; error 49 us; skew 500 ppm
I20260928 06:35:35.397528 10081 webserver.cc:460] Webserver started at http://127.5.22.2:42187/ using document root <none> and password file <none>
I20260928 06:35:35.398099 10081 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:35.398187 10081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:35.398535 10081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:35.400620 10081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "35ea047f222441ed986cf3d7ceb17de3"
format_stamp: "Formatted at 2026-09-28 06:35:35 on dist-test-slave-j0qf"
server_key: "322746c121888abe0d4b8ba69c726410"
server_key_iv: "2ff6bece860e0cedc1ce2491a02a8325"
server_key_version: "encryptionkey@0"
I20260928 06:35:35.401263 10081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "35ea047f222441ed986cf3d7ceb17de3"
format_stamp: "Formatted at 2026-09-28 06:35:35 on dist-test-slave-j0qf"
server_key: "322746c121888abe0d4b8ba69c726410"
server_key_iv: "2ff6bece860e0cedc1ce2491a02a8325"
server_key_version: "encryptionkey@0"
I20260928 06:35:35.405048 10081 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 06:35:35.407495 10095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:35.408700 10081 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:35:35.408836 10081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "35ea047f222441ed986cf3d7ceb17de3"
format_stamp: "Formatted at 2026-09-28 06:35:35 on dist-test-slave-j0qf"
server_key: "322746c121888abe0d4b8ba69c726410"
server_key_iv: "2ff6bece860e0cedc1ce2491a02a8325"
server_key_version: "encryptionkey@0"
I20260928 06:35:35.408963 10081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:35.413473 10081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:35.414041 10081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:35.414216 10081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:35.414770 10081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:35.415623 10081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:35.415689 10081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:35.415752 10081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:35.415791 10081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:35.424252 10081 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:34805
I20260928 06:35:35.424294 10210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:34805 every 8 connection(s)
I20260928 06:35:35.425119 10081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:35.427717 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 10081
I20260928 06:35:35.427838 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:35.428144 5208 external_mini_cluster.cc:1428] Setting key 180d6ceb0ba2a0942761a18cb6584e3a
I20260928 06:35:35.430084 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:37371
--builtin_ntp_servers=127.5.22.20:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 06:35:35.431553 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.426989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:55655 (local address 127.5.22.62:37371)
0928 06:35:35.427130 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:35.427133 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:35.428358 (+ 1225us) server_negotiation.cc:407] Connection header received
0928 06:35:35.429702 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:35.429705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:35.429744 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:35.429832 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:35.431004 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:35.431011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:35.431017 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:35.431034 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:35.431059 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:35.431071 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:35.431170 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:35.431331 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 06:35:35.431381 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:35:35.432353 10213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.427391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:37371 (local address 127.5.22.2:55655)
0928 06:35:35.428206 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:35.428266 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:35.429515 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:35.429978 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:35.429985 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:35.430068 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:35.430755 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:35:35.430757 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:35.430795 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:35:35.430860 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:35.431183 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:35.431191 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:35.431196 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:35:35.431345 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 06:35:35.431545 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":149,"threads_started":1}
I20260928 06:35:35.433452 10211 heartbeater.cc:347] Connected to a master server at 127.5.22.62:37371
I20260928 06:35:35.433673 10211 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:35.434094 10211 heartbeater.cc:511] Master 127.5.22.62:37371 requested a full tablet report, sending...
I20260928 06:35:35.435201 9889 ts_manager.cc:194] Registered new tserver with Master: 35ea047f222441ed986cf3d7ceb17de3 (127.5.22.2:34805)
W20260928 06:35:35.539762 10214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:35.540017 10214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:35.540056 10214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:35.544114 10214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:35.544168 10214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:35.544225 10214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:35:35.548820 10214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:37371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:35.549808 10214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:35.551184 10214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:35.560873 10219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:35.560873 10220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:35.561237 10214 server_base.cc:1034] running on GCE node
W20260928 06:35:35.561268 10222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:35.561731 10214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:35.562457 10214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:35.563649 10214 hybrid_clock.cc:648] HybridClock initialized: now 1790577335563617 us; error 44 us; skew 500 ppm
I20260928 06:35:35.565721 10214 webserver.cc:460] Webserver started at http://127.5.22.3:40695/ using document root <none> and password file <none>
I20260928 06:35:35.566392 10214 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:35.566493 10214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:35.566706 10214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:35.569000 10214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "0a1c237a462a4a8fa828510c855fab1c"
format_stamp: "Formatted at 2026-09-28 06:35:35 on dist-test-slave-j0qf"
server_key: "0569af6dc26804bd8104251bf469bdf2"
server_key_iv: "2a5dc9519e90fe04437b2c5435ef512b"
server_key_version: "encryptionkey@0"
I20260928 06:35:35.569674 10214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "0a1c237a462a4a8fa828510c855fab1c"
format_stamp: "Formatted at 2026-09-28 06:35:35 on dist-test-slave-j0qf"
server_key: "0569af6dc26804bd8104251bf469bdf2"
server_key_iv: "2a5dc9519e90fe04437b2c5435ef512b"
server_key_version: "encryptionkey@0"
I20260928 06:35:35.573920 10214 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260928 06:35:35.576618 10228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:35.577939 10214 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:35.578087 10214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "0a1c237a462a4a8fa828510c855fab1c"
format_stamp: "Formatted at 2026-09-28 06:35:35 on dist-test-slave-j0qf"
server_key: "0569af6dc26804bd8104251bf469bdf2"
server_key_iv: "2a5dc9519e90fe04437b2c5435ef512b"
server_key_version: "encryptionkey@0"
I20260928 06:35:35.578218 10214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:35.582922 10214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:35.583513 10214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:35.583685 10214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:35.584209 10214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:35.585074 10214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:35.585139 10214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:35.585202 10214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:35.585247 10214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:35.594261 10214 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:34941
I20260928 06:35:35.594285 10342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:34941 every 8 connection(s)
I20260928 06:35:35.595206 10214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:35.596793 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 10214
I20260928 06:35:35.596941 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:35.597307 5208 external_mini_cluster.cc:1428] Setting key 2f438547e8422e97ab2e0f31de4397d8
I20260928 06:35:35.601294 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.597049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:56005 (local address 127.5.22.62:37371)
0928 06:35:35.597280 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:35.597285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:35.598526 (+ 1241us) server_negotiation.cc:407] Connection header received
0928 06:35:35.599771 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:35.599774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:35.599810 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:35.599877 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:35.600925 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:35.600928 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:35.600930 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:35.600940 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:35.600960 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:35.600967 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:35.601041 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:35.601155 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 06:35:35.601191 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 06:35:35.602134 10345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.597545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:37371 (local address 127.5.22.3:56005)
0928 06:35:35.598357 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:35.598411 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:35.599582 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:35.600037 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:35.600044 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:35.600133 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:35.600690 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:35:35.600692 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:35.600731 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:35:35.600797 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:35.601040 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:35.601046 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:35.601050 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:35:35.601179 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 06:35:35.601360 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":138,"threads_started":1}
I20260928 06:35:35.603231 10343 heartbeater.cc:347] Connected to a master server at 127.5.22.62:37371
I20260928 06:35:35.603523 10343 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:35.603921 10343 heartbeater.cc:511] Master 127.5.22.62:37371 requested a full tablet report, sending...
I20260928 06:35:35.605057 9889 ts_manager.cc:194] Registered new tserver with Master: 0a1c237a462a4a8fa828510c855fab1c (127.5.22.3:34941)
I20260928 06:35:35.611186 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:35:35.614823 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.613205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33418 (local address 127.5.22.62:37371)
0928 06:35:35.613327 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:35.613330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:35.613509 (+ 179us) server_negotiation.cc:407] Connection header received
0928 06:35:35.613622 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:35.613624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:35.613658 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:35.613725 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:35.614432 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:35.614435 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:35.614438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:35.614450 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:35.614470 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:35.614480 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:35.614579 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:35.614654 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 06:35:35.614695 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:35:35.617612 9889 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:35:35.620679 9889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:35.632936 10356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.629224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41392 (local address 127.5.22.1:34571)
0928 06:35:35.629905 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:35.629919 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:35.630498 (+ 579us) server_negotiation.cc:407] Connection header received
0928 06:35:35.630690 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:35.630695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:35.630857 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:35.630984 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:35.632035 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:35.632051 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:35.632067 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:35.632085 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:35.632133 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:35.632178 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:35.632294 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:35.632636 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 06:35:35.632731 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":109,"threads_started":1}
I20260928 06:35:35.633117 10358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.629840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:34805 (local address 127.0.0.1:54714)
0928 06:35:35.630412 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:35.630424 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:35.630519 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:35.631735 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:35.631739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:35.631753 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:35.632365 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:35:35.632367 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:35.632380 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:35:35.632409 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:35.632791 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:35.632794 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:35.632797 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:35:35.632869 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 06:35:35.632942 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"spinlock_wait_cycles":5888,"thread_start_us":107,"threads_started":1}
I20260928 06:35:35.633477 10359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.630231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54714 (local address 127.5.22.2:34805)
0928 06:35:35.630677 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:35.630689 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:35.630709 (+ 20us) server_negotiation.cc:407] Connection header received
0928 06:35:35.630758 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:35.630762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:35.630892 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:35.631051 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:35.632543 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:35.632549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:35.632563 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:35.632576 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:35.632619 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:35.632648 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:35.632750 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:35.633301 (+ 551us) server_negotiation.cc:299] Negotiation successful
0928 06:35:35.633355 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":123,"threads_started":1}
I20260928 06:35:35.633592 10354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.629111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:34571 (local address 127.0.0.1:41392)
0928 06:35:35.630248 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:35.630303 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:35.630577 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:35.631019 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:35.631026 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:35.631113 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:35.631753 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:35:35.631755 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:35.631776 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:35:35.631849 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:35.632327 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:35.632340 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:35.632348 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 06:35:35.633404 (+ 1056us) client_negotiation.cc:240] Negotiation successful
0928 06:35:35.633446 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"thread_start_us":94,"threads_started":1}
I20260928 06:35:35.633906 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.629160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:34941 (local address 127.0.0.1:38320)
0928 06:35:35.630249 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:35.630303 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:35.630519 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:35.631751 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:35.631753 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:35.631766 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:35.632581 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:35:35.632586 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:35.632598 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:35:35.632623 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:35.633082 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:35.633089 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:35.633095 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:35:35.633755 (+ 660us) client_negotiation.cc:240] Negotiation successful
0928 06:35:35.633791 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":953,"spinlock_wait_cycles":7168,"thread_start_us":107,"threads_started":1}
I20260928 06:35:35.634276 10357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.629569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38320 (local address 127.5.22.3:34941)
0928 06:35:35.630023 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:35.630034 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:35.630427 (+ 393us) server_negotiation.cc:407] Connection header received
0928 06:35:35.630651 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:35.630657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:35.630857 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:35.630973 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:35.632763 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:35.632769 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:35.632780 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:35.632793 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:35.632837 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:35.632897 (+ 60us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:35.632989 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:35.633410 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 06:35:35.633486 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":145,"threads_started":1}
I20260928 06:35:35.636567 10143 tablet_service.cc:1467] Processing CreateTablet for tablet 0bbdd598a2e6405cafbc872265559806 (DEFAULT_TABLE table=test-table [id=b38be1f22a1a4993a86e8d854e7c4753]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:35.636559 10012 tablet_service.cc:1467] Processing CreateTablet for tablet 0bbdd598a2e6405cafbc872265559806 (DEFAULT_TABLE table=test-table [id=b38be1f22a1a4993a86e8d854e7c4753]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:35.636977 10277 tablet_service.cc:1467] Processing CreateTablet for tablet 0bbdd598a2e6405cafbc872265559806 (DEFAULT_TABLE table=test-table [id=b38be1f22a1a4993a86e8d854e7c4753]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:35.637660 10143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bbdd598a2e6405cafbc872265559806. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:35.637660 10012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bbdd598a2e6405cafbc872265559806. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:35.637826 10277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bbdd598a2e6405cafbc872265559806. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:35.645329 10362 tablet_bootstrap.cc:492] T 0bbdd598a2e6405cafbc872265559806 P 35ea047f222441ed986cf3d7ceb17de3: Bootstrap starting.
I20260928 06:35:35.645967 10360 tablet_bootstrap.cc:492] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c: Bootstrap starting.
I20260928 06:35:35.646646 10361 tablet_bootstrap.cc:492] T 0bbdd598a2e6405cafbc872265559806 P a83e94d3684f417a836302fd8d5569a2: Bootstrap starting.
I20260928 06:35:35.647545 10362 tablet_bootstrap.cc:654] T 0bbdd598a2e6405cafbc872265559806 P 35ea047f222441ed986cf3d7ceb17de3: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:35.647915 10360 tablet_bootstrap.cc:654] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:35.648409 10362 log.cc:824] T 0bbdd598a2e6405cafbc872265559806 P 35ea047f222441ed986cf3d7ceb17de3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:35.648701 10361 tablet_bootstrap.cc:654] T 0bbdd598a2e6405cafbc872265559806 P a83e94d3684f417a836302fd8d5569a2: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:35.648689 10360 log.cc:824] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:35.649473 10361 log.cc:824] T 0bbdd598a2e6405cafbc872265559806 P a83e94d3684f417a836302fd8d5569a2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:35.650677 10362 tablet_bootstrap.cc:492] T 0bbdd598a2e6405cafbc872265559806 P 35ea047f222441ed986cf3d7ceb17de3: No bootstrap required, opened a new log
I20260928 06:35:35.650784 10360 tablet_bootstrap.cc:492] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c: No bootstrap required, opened a new log
I20260928 06:35:35.650920 10362 ts_tablet_manager.cc:1397] T 0bbdd598a2e6405cafbc872265559806 P 35ea047f222441ed986cf3d7ceb17de3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 06:35:35.650969 10360 ts_tablet_manager.cc:1397] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 06:35:35.651322 10361 tablet_bootstrap.cc:492] T 0bbdd598a2e6405cafbc872265559806 P a83e94d3684f417a836302fd8d5569a2: No bootstrap required, opened a new log
I20260928 06:35:35.651486 10361 ts_tablet_manager.cc:1397] T 0bbdd598a2e6405cafbc872265559806 P a83e94d3684f417a836302fd8d5569a2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 06:35:35.654584 10362 raft_consensus.cc:348] T 0bbdd598a2e6405cafbc872265559806 P 35ea047f222441ed986cf3d7ceb17de3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83e94d3684f417a836302fd8d5569a2" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34571 } } peers { permanent_uuid: "0a1c237a462a4a8fa828510c855fab1c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34941 } } peers { permanent_uuid: "35ea047f222441ed986cf3d7ceb17de3" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34805 } }
I20260928 06:35:35.654935 10362 raft_consensus.cc:374] T 0bbdd598a2e6405cafbc872265559806 P 35ea047f222441ed986cf3d7ceb17de3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:35.655000 10362 raft_consensus.cc:729] T 0bbdd598a2e6405cafbc872265559806 P 35ea047f222441ed986cf3d7ceb17de3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ea047f222441ed986cf3d7ceb17de3, State: Initialized, Role: FOLLOWER
I20260928 06:35:35.655496 10362 consensus_queue.cc:260] T 0bbdd598a2e6405cafbc872265559806 P 35ea047f222441ed986cf3d7ceb17de3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83e94d3684f417a836302fd8d5569a2" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34571 } } peers { permanent_uuid: "0a1c237a462a4a8fa828510c855fab1c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34941 } } peers { permanent_uuid: "35ea047f222441ed986cf3d7ceb17de3" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34805 } }
I20260928 06:35:35.655572 10360 raft_consensus.cc:348] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83e94d3684f417a836302fd8d5569a2" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34571 } } peers { permanent_uuid: "0a1c237a462a4a8fa828510c855fab1c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34941 } } peers { permanent_uuid: "35ea047f222441ed986cf3d7ceb17de3" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34805 } }
I20260928 06:35:35.655827 10360 raft_consensus.cc:374] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:35.655884 10360 raft_consensus.cc:729] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a1c237a462a4a8fa828510c855fab1c, State: Initialized, Role: FOLLOWER
I20260928 06:35:35.655814 10361 raft_consensus.cc:348] T 0bbdd598a2e6405cafbc872265559806 P a83e94d3684f417a836302fd8d5569a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83e94d3684f417a836302fd8d5569a2" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34571 } } peers { permanent_uuid: "0a1c237a462a4a8fa828510c855fab1c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34941 } } peers { permanent_uuid: "35ea047f222441ed986cf3d7ceb17de3" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34805 } }
I20260928 06:35:35.656042 10361 raft_consensus.cc:374] T 0bbdd598a2e6405cafbc872265559806 P a83e94d3684f417a836302fd8d5569a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:35.656111 10361 raft_consensus.cc:729] T 0bbdd598a2e6405cafbc872265559806 P a83e94d3684f417a836302fd8d5569a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a83e94d3684f417a836302fd8d5569a2, State: Initialized, Role: FOLLOWER
I20260928 06:35:35.656318 10360 consensus_queue.cc:260] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83e94d3684f417a836302fd8d5569a2" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34571 } } peers { permanent_uuid: "0a1c237a462a4a8fa828510c855fab1c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34941 } } peers { permanent_uuid: "35ea047f222441ed986cf3d7ceb17de3" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34805 } }
I20260928 06:35:35.656458 10361 consensus_queue.cc:260] T 0bbdd598a2e6405cafbc872265559806 P a83e94d3684f417a836302fd8d5569a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83e94d3684f417a836302fd8d5569a2" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34571 } } peers { permanent_uuid: "0a1c237a462a4a8fa828510c855fab1c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34941 } } peers { permanent_uuid: "35ea047f222441ed986cf3d7ceb17de3" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34805 } }
I20260928 06:35:35.656764 10362 ts_tablet_manager.cc:1428] T 0bbdd598a2e6405cafbc872265559806 P 35ea047f222441ed986cf3d7ceb17de3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 06:35:35.657109 10343 heartbeater.cc:503] Master 127.5.22.62:37371 was elected leader, sending a full tablet report...
I20260928 06:35:35.657109 10211 heartbeater.cc:503] Master 127.5.22.62:37371 was elected leader, sending a full tablet report...
I20260928 06:35:35.657266 10361 ts_tablet_manager.cc:1428] T 0bbdd598a2e6405cafbc872265559806 P a83e94d3684f417a836302fd8d5569a2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 06:35:35.657267 10360 ts_tablet_manager.cc:1428] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260928 06:35:35.657605 10078 heartbeater.cc:503] Master 127.5.22.62:37371 was elected leader, sending a full tablet report...
W20260928 06:35:35.675987 10212 tablet.cc:2369] T 0bbdd598a2e6405cafbc872265559806 P 35ea047f222441ed986cf3d7ceb17de3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:35:35.754693 10079 tablet.cc:2369] T 0bbdd598a2e6405cafbc872265559806 P a83e94d3684f417a836302fd8d5569a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:35:35.846174 10344 tablet.cc:2369] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:35:35.856189 10367 raft_consensus.cc:482] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:35.856454 10367 raft_consensus.cc:504] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83e94d3684f417a836302fd8d5569a2" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34571 } } peers { permanent_uuid: "0a1c237a462a4a8fa828510c855fab1c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34941 } } peers { permanent_uuid: "35ea047f222441ed986cf3d7ceb17de3" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34805 } }
I20260928 06:35:35.857661 10367 leader_election.cc:288] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a83e94d3684f417a836302fd8d5569a2 (127.5.22.1:34571), 35ea047f222441ed986cf3d7ceb17de3 (127.5.22.2:34805)
I20260928 06:35:35.859866 10345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.857985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:34571 (local address 127.5.22.3:51085)
0928 06:35:35.858114 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:35.858151 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:35.858262 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:35.858764 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:35.858767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:35.858782 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:35.859259 (+ 477us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:35:35.859261 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:35.859269 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:35:35.859288 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:35.859580 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:35.859583 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:35.859585 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:35:35.859676 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 06:35:35.859716 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 06:35:35.859875 10359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.858462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:56177 (local address 127.5.22.2:34805)
0928 06:35:35.858591 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:35.858595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:35.858767 (+ 172us) server_negotiation.cc:407] Connection header received
0928 06:35:35.858891 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:35.858895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:35.858937 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:35.859019 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:35.859538 (+ 519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:35.859543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:35.859546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:35.859560 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:35.859579 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:35.859588 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:35.859675 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:35.859742 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 06:35:35.859775 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:35:35.859867 10356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.858087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:51085 (local address 127.5.22.1:34571)
0928 06:35:35.858243 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:35.858248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:35.858265 (+ 17us) server_negotiation.cc:407] Connection header received
0928 06:35:35.858585 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:35.858590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:35.858633 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:35.858742 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:35.859422 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:35.859425 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:35.859427 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:35.859438 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:35.859479 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:35.859491 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:35.859587 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:35.859683 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 06:35:35.859729 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 06:35:35.859889 10369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.858299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:34805 (local address 127.5.22.3:56177)
0928 06:35:35.858670 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:35.858681 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:35.858792 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:35.859026 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:35.859027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:35.859038 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:35.859376 (+ 338us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:35:35.859380 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:35.859390 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:35:35.859413 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:35.859669 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:35.859671 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:35.859674 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:35:35.859743 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:35:35.859767 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":149,"threads_started":1}
I20260928 06:35:35.860611 10164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bbdd598a2e6405cafbc872265559806" candidate_uuid: "0a1c237a462a4a8fa828510c855fab1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ea047f222441ed986cf3d7ceb17de3" is_pre_election: true
I20260928 06:35:35.860610 10032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bbdd598a2e6405cafbc872265559806" candidate_uuid: "0a1c237a462a4a8fa828510c855fab1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a83e94d3684f417a836302fd8d5569a2" is_pre_election: true
I20260928 06:35:35.860917 10032 raft_consensus.cc:2454] T 0bbdd598a2e6405cafbc872265559806 P a83e94d3684f417a836302fd8d5569a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a1c237a462a4a8fa828510c855fab1c in term 0.
I20260928 06:35:35.860917 10164 raft_consensus.cc:2454] T 0bbdd598a2e6405cafbc872265559806 P 35ea047f222441ed986cf3d7ceb17de3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a1c237a462a4a8fa828510c855fab1c in term 0.
I20260928 06:35:35.861378 10232 leader_election.cc:302] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a1c237a462a4a8fa828510c855fab1c, 35ea047f222441ed986cf3d7ceb17de3; no voters:
I20260928 06:35:35.861620 10367 raft_consensus.cc:2781] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:35.861716 10367 raft_consensus.cc:482] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:35.861773 10367 raft_consensus.cc:3037] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:35.862864 10367 raft_consensus.cc:504] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83e94d3684f417a836302fd8d5569a2" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34571 } } peers { permanent_uuid: "0a1c237a462a4a8fa828510c855fab1c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34941 } } peers { permanent_uuid: "35ea047f222441ed986cf3d7ceb17de3" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34805 } }
I20260928 06:35:35.863193 10367 leader_election.cc:288] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c [CANDIDATE]: Term 1 election: Requested vote from peers a83e94d3684f417a836302fd8d5569a2 (127.5.22.1:34571), 35ea047f222441ed986cf3d7ceb17de3 (127.5.22.2:34805)
I20260928 06:35:35.863544 10032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bbdd598a2e6405cafbc872265559806" candidate_uuid: "0a1c237a462a4a8fa828510c855fab1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a83e94d3684f417a836302fd8d5569a2"
I20260928 06:35:35.863581 10164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bbdd598a2e6405cafbc872265559806" candidate_uuid: "0a1c237a462a4a8fa828510c855fab1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ea047f222441ed986cf3d7ceb17de3"
I20260928 06:35:35.863673 10032 raft_consensus.cc:3037] T 0bbdd598a2e6405cafbc872265559806 P a83e94d3684f417a836302fd8d5569a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:35.863703 10164 raft_consensus.cc:3037] T 0bbdd598a2e6405cafbc872265559806 P 35ea047f222441ed986cf3d7ceb17de3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:35.864787 10164 raft_consensus.cc:2454] T 0bbdd598a2e6405cafbc872265559806 P 35ea047f222441ed986cf3d7ceb17de3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a1c237a462a4a8fa828510c855fab1c in term 1.
I20260928 06:35:35.864812 10032 raft_consensus.cc:2454] T 0bbdd598a2e6405cafbc872265559806 P a83e94d3684f417a836302fd8d5569a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a1c237a462a4a8fa828510c855fab1c in term 1.
I20260928 06:35:35.865116 10232 leader_election.cc:302] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a1c237a462a4a8fa828510c855fab1c, a83e94d3684f417a836302fd8d5569a2; no voters:
I20260928 06:35:35.865423 10367 raft_consensus.cc:2781] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:35.865746 10367 raft_consensus.cc:686] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c [term 1 LEADER]: Becoming Leader. State: Replica: 0a1c237a462a4a8fa828510c855fab1c, State: Running, Role: LEADER
I20260928 06:35:35.866106 10367 consensus_queue.cc:237] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83e94d3684f417a836302fd8d5569a2" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34571 } } peers { permanent_uuid: "0a1c237a462a4a8fa828510c855fab1c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34941 } } peers { permanent_uuid: "35ea047f222441ed986cf3d7ceb17de3" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34805 } }
I20260928 06:35:35.869791 9888 catalog_manager.cc:5495] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a1c237a462a4a8fa828510c855fab1c (127.5.22.3). New cstate: current_term: 1 leader_uuid: "0a1c237a462a4a8fa828510c855fab1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83e94d3684f417a836302fd8d5569a2" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a1c237a462a4a8fa828510c855fab1c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35ea047f222441ed986cf3d7ceb17de3" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34805 } health_report { overall_health: UNKNOWN } } }
I20260928 06:35:35.933947 10357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.932207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38324 (local address 127.5.22.3:34941)
0928 06:35:35.932469 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:35.932473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:35.932487 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:35:35.932572 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:35.932574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:35.932639 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:35.932745 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:35.933505 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:35.933514 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:35.933516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:35.933527 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:35.933544 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:35.933553 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:35.933648 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:35.933753 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 06:35:35.933794 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 06:35:35.940589 10032 raft_consensus.cc:1261] T 0bbdd598a2e6405cafbc872265559806 P a83e94d3684f417a836302fd8d5569a2 [term 1 FOLLOWER]: Refusing update from remote peer 0a1c237a462a4a8fa828510c855fab1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:35.940588 10164 raft_consensus.cc:1261] T 0bbdd598a2e6405cafbc872265559806 P 35ea047f222441ed986cf3d7ceb17de3 [term 1 FOLLOWER]: Refusing update from remote peer 0a1c237a462a4a8fa828510c855fab1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:35.941429 10367 consensus_queue.cc:1035] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "35ea047f222441ed986cf3d7ceb17de3" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:35.941674 10370 consensus_queue.cc:1035] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a83e94d3684f417a836302fd8d5569a2" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:35.951061 10375 mvcc.cc:205] Tried to move back new op lower bound from 7334204768004718592 to 7334204767710674944. Current Snapshot: MvccSnapshot[applied={T|T < 7334204768004718592}]
I20260928 06:35:35.951242 10374 mvcc.cc:205] Tried to move back new op lower bound from 7334204768004718592 to 7334204767710674944. Current Snapshot: MvccSnapshot[applied={T|T < 7334204768004718592}]
I20260928 06:35:35.957082 9888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:35.957263 9888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:35.960052 9888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0690e7b46051471788e2887e1220aa4b: Sending DeleteTablet for 3 replicas of tablet 0bbdd598a2e6405cafbc872265559806
I20260928 06:35:35.960963 10277 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bbdd598a2e6405cafbc872265559806 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:35 UTC) from {username='slave'} at 127.0.0.1:38320
I20260928 06:35:35.961037 10012 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bbdd598a2e6405cafbc872265559806 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:35 UTC) from {username='slave'} at 127.0.0.1:41392
I20260928 06:35:35.961489 10383 tablet_replica.cc:330] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c: stopping tablet replica
I20260928 06:35:35.961515 10143 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bbdd598a2e6405cafbc872265559806 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:35 UTC) from {username='slave'} at 127.0.0.1:54714
I20260928 06:35:35.961752 10384 tablet_replica.cc:330] T 0bbdd598a2e6405cafbc872265559806 P a83e94d3684f417a836302fd8d5569a2: stopping tablet replica
I20260928 06:35:35.961992 10383 raft_consensus.cc:2229] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:35:35.962069 10384 raft_consensus.cc:2229] T 0bbdd598a2e6405cafbc872265559806 P a83e94d3684f417a836302fd8d5569a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:35.962440 10384 raft_consensus.cc:2258] T 0bbdd598a2e6405cafbc872265559806 P a83e94d3684f417a836302fd8d5569a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:35.962604 10383 raft_consensus.cc:2258] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:35.962764 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 9949
I20260928 06:35:35.963845 10385 tablet_replica.cc:330] T 0bbdd598a2e6405cafbc872265559806 P 35ea047f222441ed986cf3d7ceb17de3: stopping tablet replica
I20260928 06:35:35.964147 10385 raft_consensus.cc:2229] T 0bbdd598a2e6405cafbc872265559806 P 35ea047f222441ed986cf3d7ceb17de3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:35.964442 10385 raft_consensus.cc:2258] T 0bbdd598a2e6405cafbc872265559806 P 35ea047f222441ed986cf3d7ceb17de3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:35.964993 10383 ts_tablet_manager.cc:1905] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:35.966015 10385 ts_tablet_manager.cc:1905] T 0bbdd598a2e6405cafbc872265559806 P 35ea047f222441ed986cf3d7ceb17de3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:35.968653 10385 ts_tablet_manager.cc:1918] T 0bbdd598a2e6405cafbc872265559806 P 35ea047f222441ed986cf3d7ceb17de3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:35.968762 10385 log.cc:1196] T 0bbdd598a2e6405cafbc872265559806 P 35ea047f222441ed986cf3d7ceb17de3: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-1/wal/wals/0bbdd598a2e6405cafbc872265559806
I20260928 06:35:35.968981 10383 ts_tablet_manager.cc:1918] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:35.969082 10385 ts_tablet_manager.cc:1939] T 0bbdd598a2e6405cafbc872265559806 P 35ea047f222441ed986cf3d7ceb17de3: Deleting consensus metadata
I20260928 06:35:35.969182 10383 log.cc:1196] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/0bbdd598a2e6405cafbc872265559806
I20260928 06:35:35.969564 10383 ts_tablet_manager.cc:1939] T 0bbdd598a2e6405cafbc872265559806 P 0a1c237a462a4a8fa828510c855fab1c: Deleting consensus metadata
I20260928 06:35:35.970463 9875 catalog_manager.cc:4841] TS 35ea047f222441ed986cf3d7ceb17de3 (127.5.22.2:34805): tablet 0bbdd598a2e6405cafbc872265559806 (table test-table [id=b38be1f22a1a4993a86e8d854e7c4753]) successfully deleted
I20260928 06:35:35.970806 9873 catalog_manager.cc:4841] TS 0a1c237a462a4a8fa828510c855fab1c (127.5.22.3:34941): tablet 0bbdd598a2e6405cafbc872265559806 (table test-table [id=b38be1f22a1a4993a86e8d854e7c4753]) successfully deleted
W20260928 06:35:35.971846 9875 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.22.1:34571 (error 108)
I20260928 06:35:35.972080 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 10081
I20260928 06:35:35.973073 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.972703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:34571 (local address 127.0.0.1:41394)
0928 06:35:35.972899 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:35.972976 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.1:34571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260928 06:35:35.973330 9875 catalog_manager.cc:4584] TS a83e94d3684f417a836302fd8d5569a2 (127.5.22.1:34571): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0bbdd598a2e6405cafbc872265559806: Network error: Client connection negotiation failed: client connection to 127.5.22.1:34571: connect: Connection refused (error 111)
I20260928 06:35:35.982079 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 10214
I20260928 06:35:35.987759 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 9856
2026-09-28T06:35:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1174 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 06:35:36.009840 5208 mini_oidc.cc:152] Starting JWKS server
I20260928 06:35:36.011559 5208 webserver.cc:460] Webserver started at https://127.0.0.1:42471/ using document root <none> and password file <none>
I20260928 06:35:36.011756 5208 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 06:35:36.012030 5208 webserver.cc:460] Webserver started at http://127.0.0.1:38215/ using document root <none> and password file <none>
2026-09-28T06:35:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:36Z Disabled control of system clock
I20260928 06:35:36.021299 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:36423
--webserver_interface=127.5.22.62
--webserver_port=0
--builtin_ntp_servers=127.5.22.20:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:36423
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42471/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577308481727-5208-0/testchainca.cert with env {}
W20260928 06:35:36.132804 10392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:36.133078 10392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:36.133118 10392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:36.137157 10392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:36.137209 10392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:36.137225 10392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:36.137241 10392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:36.137257 10392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:35:36.137274 10392 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:35:36.137286 10392 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42471/jwks.json
I20260928 06:35:36.142485 10392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:36423
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:36423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42471/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577308481727-5208-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:36.143446 10392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:36.144399 10392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:35:36.152971 10392 server_base.cc:1034] running on GCE node
W20260928 06:35:36.152932 10398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:36.152958 10397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:36.152936 10400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:36.153679 10392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:36.154778 10392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:36.155961 10392 hybrid_clock.cc:648] HybridClock initialized: now 1790577336155947 us; error 28 us; skew 500 ppm
I20260928 06:35:36.157816 10392 webserver.cc:460] Webserver started at http://127.5.22.62:40917/ using document root <none> and password file <none>
I20260928 06:35:36.158427 10392 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:36.158515 10392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:36.158762 10392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:36.160732 10392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "c8a118953c494faf85bbed0b08294b8c"
format_stamp: "Formatted at 2026-09-28 06:35:36 on dist-test-slave-j0qf"
server_key: "c18a213088c5f3c88b198da17f7ae940"
server_key_iv: "5b7515e40c26766a35d8adf3839ad2d3"
server_key_version: "encryptionkey@0"
I20260928 06:35:36.161283 10392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "c8a118953c494faf85bbed0b08294b8c"
format_stamp: "Formatted at 2026-09-28 06:35:36 on dist-test-slave-j0qf"
server_key: "c18a213088c5f3c88b198da17f7ae940"
server_key_iv: "5b7515e40c26766a35d8adf3839ad2d3"
server_key_version: "encryptionkey@0"
I20260928 06:35:36.164824 10392 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 06:35:36.167143 10406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:36.168408 10392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:35:36.168555 10392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "c8a118953c494faf85bbed0b08294b8c"
format_stamp: "Formatted at 2026-09-28 06:35:36 on dist-test-slave-j0qf"
server_key: "c18a213088c5f3c88b198da17f7ae940"
server_key_iv: "5b7515e40c26766a35d8adf3839ad2d3"
server_key_version: "encryptionkey@0"
I20260928 06:35:36.168664 10392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:36.193893 10392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:36.194648 10392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:36.194842 10392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:36.202131 10392 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:36423
I20260928 06:35:36.202147 10459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:36423 every 8 connection(s)
I20260928 06:35:36.203161 10392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:36.206679 10460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:36.207291 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 10392
I20260928 06:35:36.207442 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:35:36.207707 5208 external_mini_cluster.cc:1428] Setting key eba00b1aa2efd9e2a133a78b5550c36a
I20260928 06:35:36.213336 10460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8a118953c494faf85bbed0b08294b8c: Bootstrap starting.
I20260928 06:35:36.215891 10460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8a118953c494faf85bbed0b08294b8c: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:36.216815 10460 log.cc:824] T 00000000000000000000000000000000 P c8a118953c494faf85bbed0b08294b8c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:36.217453 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:36.209168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41824 (local address 127.5.22.62:36423)
0928 06:35:36.209866 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:36.209881 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:36.209917 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:35:36.210645 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:36.210664 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:36.211081 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:36.211507 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:36.211962 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:36.212924 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:36.213849 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:36.214365 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:36.214546 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:36.215540 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:36.215545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:36.215558 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:36.215953 (+ 395us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:36.216041 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:36.216069 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:36.216180 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:36.216327 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 06:35:36.216599 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":130,"threads_started":1}
I20260928 06:35:36.219197 10460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8a118953c494faf85bbed0b08294b8c: No bootstrap required, opened a new log
I20260928 06:35:36.220307 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:36.223477 10460 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8a118953c494faf85bbed0b08294b8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8a118953c494faf85bbed0b08294b8c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 36423 } }
I20260928 06:35:36.223750 10460 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8a118953c494faf85bbed0b08294b8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:36.223824 10460 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8a118953c494faf85bbed0b08294b8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a118953c494faf85bbed0b08294b8c, State: Initialized, Role: FOLLOWER
I20260928 06:35:36.224274 10460 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8a118953c494faf85bbed0b08294b8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8a118953c494faf85bbed0b08294b8c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 36423 } }
I20260928 06:35:36.224414 10460 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8a118953c494faf85bbed0b08294b8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:36.224481 10460 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8a118953c494faf85bbed0b08294b8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:36.224592 10460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8a118953c494faf85bbed0b08294b8c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:36.225513 10460 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8a118953c494faf85bbed0b08294b8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8a118953c494faf85bbed0b08294b8c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 36423 } }
I20260928 06:35:36.225862 10460 leader_election.cc:302] T 00000000000000000000000000000000 P c8a118953c494faf85bbed0b08294b8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8a118953c494faf85bbed0b08294b8c; no voters:
I20260928 06:35:36.226202 10460 leader_election.cc:288] T 00000000000000000000000000000000 P c8a118953c494faf85bbed0b08294b8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:36.226312 10465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8a118953c494faf85bbed0b08294b8c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:36.226595 10465 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8a118953c494faf85bbed0b08294b8c [term 1 LEADER]: Becoming Leader. State: Replica: c8a118953c494faf85bbed0b08294b8c, State: Running, Role: LEADER
I20260928 06:35:36.226913 10465 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8a118953c494faf85bbed0b08294b8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8a118953c494faf85bbed0b08294b8c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 36423 } }
I20260928 06:35:36.227464 10460 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8a118953c494faf85bbed0b08294b8c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:36.228515 10466 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8a118953c494faf85bbed0b08294b8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8a118953c494faf85bbed0b08294b8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8a118953c494faf85bbed0b08294b8c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 36423 } } }
I20260928 06:35:36.228663 10466 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8a118953c494faf85bbed0b08294b8c [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:36.229072 10474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:36.229179 10467 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8a118953c494faf85bbed0b08294b8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8a118953c494faf85bbed0b08294b8c. Latest consensus state: current_term: 1 leader_uuid: "c8a118953c494faf85bbed0b08294b8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8a118953c494faf85bbed0b08294b8c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 36423 } } }
I20260928 06:35:36.229265 10467 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8a118953c494faf85bbed0b08294b8c [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:36.232266 10474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:36.237937 10474 catalog_manager.cc:1347] Generated new cluster ID: 1f5866df1b634a34bd34b4aa35f1caea
I20260928 06:35:36.238015 10474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:36.244109 10474 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:36.244944 10474 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:36.251498 10474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8a118953c494faf85bbed0b08294b8c: Generated new TSK 0
I20260928 06:35:36.252171 10474 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 06:35:36.287910 10490 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 06:35:36.288038 10490 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 06:35:36.288568 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:36.271076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41826 (local address 127.5.22.62:36423)
0928 06:35:36.271219 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:36.271224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:36.271444 (+ 220us) server_negotiation.cc:407] Connection header received
0928 06:35:36.271685 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:36.271688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:36.271736 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:36.271816 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:35:36.272182 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:36.272765 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:36.274055 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:36.274279 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:36.274437 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:36.274796 (+ 359us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:35:36.288221 (+ 13425us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42471/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 06:35:36.288453 (+ 232us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41826: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42471/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 06:35:36.288647 10463 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41826: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42471/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 06:35:36.290048 10489 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:36.270996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:36423 (local address 127.0.0.1:41826)
0928 06:35:36.271354 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:36.271369 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:36.271509 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:36.271856 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:36.271859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:36.271867 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:35:36.272015 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:36.272022 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:36.272873 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:36.272877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:36.273910 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:36.273921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:36.274435 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:36.274456 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:36.274555 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:36.274619 (+ 64us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:35:36.288457 (+ 13838us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42471/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 06:35:36.288724 (+ 267us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.22.62:36423: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42471/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":122,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.5.22.62:36423: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42471/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 06:35:36.290822 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 10392
2026-09-28T06:35:36Z chronyd exiting
I20260928 06:35:36.328181 5208 test_util.cc:175] -----------------------------------------------
I20260928 06:35:36.328269 5208 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577308481727-5208-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (319 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 06:35:36.329330 5208 mini_oidc.cc:152] Starting JWKS server
I20260928 06:35:36.330210 5208 webserver.cc:460] Webserver started at https://127.0.0.1:42507/ using document root <none> and password file <none>
I20260928 06:35:36.330353 5208 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 06:35:36.330646 5208 webserver.cc:460] Webserver started at http://127.0.0.1:34233/ using document root <none> and password file <none>
2026-09-28T06:35:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:36Z Disabled control of system clock
I20260928 06:35:36.339778 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:46717
--webserver_interface=127.5.22.62
--webserver_port=0
--builtin_ntp_servers=127.5.22.20:43997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:46717
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42507/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577308481727-5208-0/testchainca.cert with env {}
W20260928 06:35:36.448231 10496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:36.448490 10496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:36.448537 10496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:36.452497 10496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:36.452554 10496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:36.452570 10496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:36.452587 10496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:36.452602 10496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:35:36.452617 10496 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:35:36.452631 10496 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42507/jwks.json
I20260928 06:35:36.457859 10496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:43997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:46717
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:46717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42507/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577308481727-5208-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:36.458855 10496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:36.459810 10496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:36.468145 10501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:36.468217 10504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:36.468171 10502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:36.468454 10496 server_base.cc:1034] running on GCE node
I20260928 06:35:36.469043 10496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:36.470073 10496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:36.471220 10496 hybrid_clock.cc:648] HybridClock initialized: now 1790577336471190 us; error 47 us; skew 500 ppm
I20260928 06:35:36.472935 10496 webserver.cc:460] Webserver started at http://127.5.22.62:43277/ using document root <none> and password file <none>
I20260928 06:35:36.473460 10496 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:36.473519 10496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:36.473671 10496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:36.475625 10496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "b506a1474ef3481bbae80db28674a8da"
format_stamp: "Formatted at 2026-09-28 06:35:36 on dist-test-slave-j0qf"
server_key: "63d15ffc4660443495ee86288b2f1cef"
server_key_iv: "686239a0c12e10cd97e63f45e9558882"
server_key_version: "encryptionkey@0"
I20260928 06:35:36.476130 10496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "b506a1474ef3481bbae80db28674a8da"
format_stamp: "Formatted at 2026-09-28 06:35:36 on dist-test-slave-j0qf"
server_key: "63d15ffc4660443495ee86288b2f1cef"
server_key_iv: "686239a0c12e10cd97e63f45e9558882"
server_key_version: "encryptionkey@0"
I20260928 06:35:36.479579 10496 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:35:36.481818 10510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:36.483052 10496 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:36.483181 10496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "b506a1474ef3481bbae80db28674a8da"
format_stamp: "Formatted at 2026-09-28 06:35:36 on dist-test-slave-j0qf"
server_key: "63d15ffc4660443495ee86288b2f1cef"
server_key_iv: "686239a0c12e10cd97e63f45e9558882"
server_key_version: "encryptionkey@0"
I20260928 06:35:36.483286 10496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:36.497679 10496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:36.498428 10496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:36.498615 10496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:36.505415 10496 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:46717
I20260928 06:35:36.505427 10562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:46717 every 8 connection(s)
I20260928 06:35:36.506343 10496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:36.509692 10563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:36.515075 10563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b506a1474ef3481bbae80db28674a8da: Bootstrap starting.
I20260928 06:35:36.515202 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 10496
I20260928 06:35:36.515334 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:35:36.515663 5208 external_mini_cluster.cc:1428] Setting key 49fb75d66c4a6e1ebfc4ac02a10536c5
I20260928 06:35:36.517539 10563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b506a1474ef3481bbae80db28674a8da: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:36.518297 10563 log.cc:824] T 00000000000000000000000000000000 P b506a1474ef3481bbae80db28674a8da: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:36.520167 10563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b506a1474ef3481bbae80db28674a8da: No bootstrap required, opened a new log
I20260928 06:35:36.523530 10563 raft_consensus.cc:348] T 00000000000000000000000000000000 P b506a1474ef3481bbae80db28674a8da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b506a1474ef3481bbae80db28674a8da" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 46717 } }
I20260928 06:35:36.523766 10563 raft_consensus.cc:374] T 00000000000000000000000000000000 P b506a1474ef3481bbae80db28674a8da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:36.523838 10563 raft_consensus.cc:729] T 00000000000000000000000000000000 P b506a1474ef3481bbae80db28674a8da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b506a1474ef3481bbae80db28674a8da, State: Initialized, Role: FOLLOWER
I20260928 06:35:36.524261 10563 consensus_queue.cc:260] T 00000000000000000000000000000000 P b506a1474ef3481bbae80db28674a8da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b506a1474ef3481bbae80db28674a8da" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 46717 } }
I20260928 06:35:36.524427 10563 raft_consensus.cc:388] T 00000000000000000000000000000000 P b506a1474ef3481bbae80db28674a8da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:36.524509 10563 raft_consensus.cc:482] T 00000000000000000000000000000000 P b506a1474ef3481bbae80db28674a8da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:36.524623 10563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b506a1474ef3481bbae80db28674a8da [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:36.524887 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:36.516897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43730 (local address 127.5.22.62:46717)
0928 06:35:36.517556 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:36.517579 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:36.517625 (+ 46us) server_negotiation.cc:407] Connection header received
0928 06:35:36.518294 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:36.518310 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:36.518790 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:36.519120 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:36.519536 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:36.520541 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:36.521426 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:36.521917 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:36.522098 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:36.522957 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:36.522962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:36.522974 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:36.523349 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:36.523406 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:36.523426 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:36.523531 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:36.523647 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 06:35:36.523954 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":121,"threads_started":1}
I20260928 06:35:36.525570 10563 raft_consensus.cc:504] T 00000000000000000000000000000000 P b506a1474ef3481bbae80db28674a8da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b506a1474ef3481bbae80db28674a8da" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 46717 } }
I20260928 06:35:36.525897 10563 leader_election.cc:302] T 00000000000000000000000000000000 P b506a1474ef3481bbae80db28674a8da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b506a1474ef3481bbae80db28674a8da; no voters:
I20260928 06:35:36.526185 10563 leader_election.cc:288] T 00000000000000000000000000000000 P b506a1474ef3481bbae80db28674a8da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:36.526278 10568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b506a1474ef3481bbae80db28674a8da [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:36.526832 10568 raft_consensus.cc:686] T 00000000000000000000000000000000 P b506a1474ef3481bbae80db28674a8da [term 1 LEADER]: Becoming Leader. State: Replica: b506a1474ef3481bbae80db28674a8da, State: Running, Role: LEADER
I20260928 06:35:36.527169 10568 consensus_queue.cc:237] T 00000000000000000000000000000000 P b506a1474ef3481bbae80db28674a8da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b506a1474ef3481bbae80db28674a8da" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 46717 } }
I20260928 06:35:36.527433 10563 sys_catalog.cc:564] T 00000000000000000000000000000000 P b506a1474ef3481bbae80db28674a8da [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:36.527623 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:36.528746 10569 sys_catalog.cc:455] T 00000000000000000000000000000000 P b506a1474ef3481bbae80db28674a8da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b506a1474ef3481bbae80db28674a8da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b506a1474ef3481bbae80db28674a8da" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 46717 } } }
I20260928 06:35:36.528889 10569 sys_catalog.cc:458] T 00000000000000000000000000000000 P b506a1474ef3481bbae80db28674a8da [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:36.529214 10570 sys_catalog.cc:455] T 00000000000000000000000000000000 P b506a1474ef3481bbae80db28674a8da [sys.catalog]: SysCatalogTable state changed. Reason: New leader b506a1474ef3481bbae80db28674a8da. Latest consensus state: current_term: 1 leader_uuid: "b506a1474ef3481bbae80db28674a8da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b506a1474ef3481bbae80db28674a8da" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 46717 } } }
I20260928 06:35:36.529321 10570 sys_catalog.cc:458] T 00000000000000000000000000000000 P b506a1474ef3481bbae80db28674a8da [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:36.529681 10579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:36.532523 10579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:36.537953 10579 catalog_manager.cc:1347] Generated new cluster ID: 4a02d9a7533440c2b94ae4c09b5a21c8
I20260928 06:35:36.538033 10579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:36.547578 10579 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:36.548446 10579 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:36.554064 10579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b506a1474ef3481bbae80db28674a8da: Generated new TSK 0
I20260928 06:35:36.554716 10579 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 06:35:36.582167 10593 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 06:35:36.582261 10593 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 06:35:36.583029 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:36.567205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43736 (local address 127.5.22.62:46717)
0928 06:35:36.567339 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:36.567345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:36.567490 (+ 145us) server_negotiation.cc:407] Connection header received
0928 06:35:36.567639 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:36.567642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:36.567700 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:36.567808 (+ 108us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:35:36.568014 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:36.568841 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:36.569687 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:36.570040 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:36.570145 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:36.570437 (+ 292us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:35:36.582604 (+ 12167us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42507/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 06:35:36.582884 (+ 280us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43736: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42507/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 06:35:36.583103 10566 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43736: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42507/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 06:35:36.583141 10592 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:36.567115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:46717 (local address 127.0.0.1:43736)
0928 06:35:36.567405 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:36.567417 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:36.567510 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:36.567820 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:36.567822 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:36.567828 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:35:36.567921 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:36.567925 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:36.568939 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:36.568941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:36.569574 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:36.569580 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:36.570145 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:36.570161 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:36.570224 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:36.570245 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:35:36.582811 (+ 12566us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42507/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 06:35:36.582970 (+ 159us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.22.62:46717: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42507/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":100,"threads_started":1}
I20260928 06:35:36.583760 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 10496
2026-09-28T06:35:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (293 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 06:35:36.622740 5208 mini_oidc.cc:152] Starting JWKS server
I20260928 06:35:36.623576 5208 webserver.cc:460] Webserver started at https://127.0.0.1:38877/ using document root <none> and password file <none>
I20260928 06:35:36.623656 5208 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 06:35:36.623865 5208 webserver.cc:460] Webserver started at http://127.0.0.1:36691/ using document root <none> and password file <none>
2026-09-28T06:35:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:36Z Disabled control of system clock
I20260928 06:35:36.633015 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:42273
--webserver_interface=127.5.22.62
--webserver_port=0
--builtin_ntp_servers=127.5.22.20:37539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:42273
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38877/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 06:35:36.742192 10599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:36.742523 10599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:36.742564 10599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:36.746592 10599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:36.746660 10599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:36.746676 10599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:36.746693 10599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:36.746708 10599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:35:36.746726 10599 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:35:36.746739 10599 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38877/jwks.json
I20260928 06:35:36.751648 10599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:37539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:42273
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:42273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38877/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:36.752751 10599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:36.754104 10599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:36.762627 10607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:36.762636 10604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:36.762745 10599 server_base.cc:1034] running on GCE node
W20260928 06:35:36.762636 10605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:36.763464 10599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:36.764611 10599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:36.765802 10599 hybrid_clock.cc:648] HybridClock initialized: now 1790577336765788 us; error 29 us; skew 500 ppm
I20260928 06:35:36.767915 10599 webserver.cc:460] Webserver started at http://127.5.22.62:42425/ using document root <none> and password file <none>
I20260928 06:35:36.768496 10599 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:36.768591 10599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:36.768813 10599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:36.770951 10599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "a1e1b163412842ae86308c7154abf7ce"
format_stamp: "Formatted at 2026-09-28 06:35:36 on dist-test-slave-j0qf"
server_key: "84e6ba11335608c6b08e01c725b3a280"
server_key_iv: "24fc1903bc906785f55bf220e5c9537f"
server_key_version: "encryptionkey@0"
I20260928 06:35:36.771528 10599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "a1e1b163412842ae86308c7154abf7ce"
format_stamp: "Formatted at 2026-09-28 06:35:36 on dist-test-slave-j0qf"
server_key: "84e6ba11335608c6b08e01c725b3a280"
server_key_iv: "24fc1903bc906785f55bf220e5c9537f"
server_key_version: "encryptionkey@0"
I20260928 06:35:36.775673 10599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 06:35:36.778149 10613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:36.779486 10599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:35:36.779639 10599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "a1e1b163412842ae86308c7154abf7ce"
format_stamp: "Formatted at 2026-09-28 06:35:36 on dist-test-slave-j0qf"
server_key: "84e6ba11335608c6b08e01c725b3a280"
server_key_iv: "24fc1903bc906785f55bf220e5c9537f"
server_key_version: "encryptionkey@0"
I20260928 06:35:36.779759 10599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:36.795809 10599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:36.796535 10599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:36.796730 10599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:36.803929 10599 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:42273
I20260928 06:35:36.803951 10665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:42273 every 8 connection(s)
I20260928 06:35:36.804904 10599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:36.808363 10666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:36.808526 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 10599
I20260928 06:35:36.808640 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:35:36.808944 5208 external_mini_cluster.cc:1428] Setting key aecc903b197c22ec9aa42bed0f9988aa
I20260928 06:35:36.815853 10666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1e1b163412842ae86308c7154abf7ce: Bootstrap starting.
I20260928 06:35:36.818463 10666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1e1b163412842ae86308c7154abf7ce: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:36.818686 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:36.810385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56244 (local address 127.5.22.62:42273)
0928 06:35:36.811024 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:36.811041 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:36.811076 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:35:36.811794 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:36.811812 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:36.812212 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:36.812514 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:36.812966 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:36.814096 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:36.815345 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:36.815834 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:36.815959 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:36.816836 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:36.816841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:36.816853 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:36.817241 (+ 388us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:36.817297 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:36.817317 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:36.817393 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:36.817573 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 06:35:36.817862 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":133,"threads_started":1}
I20260928 06:35:36.819262 10666 log.cc:824] T 00000000000000000000000000000000 P a1e1b163412842ae86308c7154abf7ce: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:36.821239 10666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1e1b163412842ae86308c7154abf7ce: No bootstrap required, opened a new log
I20260928 06:35:36.821436 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:36.824944 10666 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1e1b163412842ae86308c7154abf7ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e1b163412842ae86308c7154abf7ce" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 42273 } }
I20260928 06:35:36.825176 10666 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1e1b163412842ae86308c7154abf7ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:36.825217 10666 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1e1b163412842ae86308c7154abf7ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1e1b163412842ae86308c7154abf7ce, State: Initialized, Role: FOLLOWER
I20260928 06:35:36.825640 10666 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1e1b163412842ae86308c7154abf7ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e1b163412842ae86308c7154abf7ce" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 42273 } }
I20260928 06:35:36.825762 10666 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1e1b163412842ae86308c7154abf7ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:36.825843 10666 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1e1b163412842ae86308c7154abf7ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:36.825961 10666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1e1b163412842ae86308c7154abf7ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:36.826990 10666 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1e1b163412842ae86308c7154abf7ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e1b163412842ae86308c7154abf7ce" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 42273 } }
I20260928 06:35:36.827373 10666 leader_election.cc:302] T 00000000000000000000000000000000 P a1e1b163412842ae86308c7154abf7ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1e1b163412842ae86308c7154abf7ce; no voters:
I20260928 06:35:36.827783 10666 leader_election.cc:288] T 00000000000000000000000000000000 P a1e1b163412842ae86308c7154abf7ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:36.827906 10671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1e1b163412842ae86308c7154abf7ce [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:36.828146 10671 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1e1b163412842ae86308c7154abf7ce [term 1 LEADER]: Becoming Leader. State: Replica: a1e1b163412842ae86308c7154abf7ce, State: Running, Role: LEADER
I20260928 06:35:36.828450 10671 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1e1b163412842ae86308c7154abf7ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e1b163412842ae86308c7154abf7ce" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 42273 } }
I20260928 06:35:36.829187 10666 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1e1b163412842ae86308c7154abf7ce [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:36.830044 10672 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1e1b163412842ae86308c7154abf7ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1e1b163412842ae86308c7154abf7ce. Latest consensus state: current_term: 1 leader_uuid: "a1e1b163412842ae86308c7154abf7ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e1b163412842ae86308c7154abf7ce" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 42273 } } }
I20260928 06:35:36.830192 10672 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1e1b163412842ae86308c7154abf7ce [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:36.830029 10673 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1e1b163412842ae86308c7154abf7ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1e1b163412842ae86308c7154abf7ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e1b163412842ae86308c7154abf7ce" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 42273 } } }
I20260928 06:35:36.830479 10673 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1e1b163412842ae86308c7154abf7ce [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:36.830768 10680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:36.834225 10680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:36.840023 10680 catalog_manager.cc:1347] Generated new cluster ID: 7e358f0b56cc47e993f09d7cb7e9ea35
I20260928 06:35:36.840096 10680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:36.849325 10680 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:36.850235 10680 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:36.860126 10680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1e1b163412842ae86308c7154abf7ce: Generated new TSK 0
I20260928 06:35:36.860913 10680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:36.879388 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:36.873519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56246 (local address 127.5.22.62:42273)
0928 06:35:36.873658 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:36.873666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:36.874614 (+ 948us) server_negotiation.cc:407] Connection header received
0928 06:35:36.874774 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:36.874776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:36.874829 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:36.874926 (+ 97us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:35:36.875181 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:36.875731 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:36.876945 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:36.877168 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:36.877241 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:36.877579 (+ 338us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:35:36.879029 (+ 1450us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38877/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 06:35:36.879252 (+ 223us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56246: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38877/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 06:35:36.879459 10669 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56246: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38877/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 06:35:36.881282 10695 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:36.873427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:42273 (local address 127.0.0.1:56246)
0928 06:35:36.874512 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:36.874527 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:36.874650 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:36.874948 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:36.874951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:36.874964 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:35:36.875080 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:36.875086 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:36.875836 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:36.875839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:36.876837 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:36.876847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:36.877259 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:36.877283 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:36.877372 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:36.877403 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:35:36.879206 (+ 1803us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38877/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 06:35:36.881105 (+ 1899us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.22.62:42273: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38877/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":990,"mutex_wait_us":1698,"thread_start_us":112,"threads_started":1}
I20260928 06:35:36.881881 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 10599
E20260928 06:35:36.890367 10696 webserver.cc:533] Webserver: sslize failed:
E20260928 06:35:36.890439 10696 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T06:35:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (298 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T06:35:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:36Z Disabled control of system clock
I20260928 06:35:36.930753 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:40921
--webserver_interface=127.5.22.62
--webserver_port=0
--builtin_ntp_servers=127.5.22.20:41645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:40921
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577308481727-5208-0/testchainca.cert with env {}
W20260928 06:35:37.044384 10700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:37.044711 10700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:37.044782 10700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:37.048943 10700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:37.049039 10700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:37.049072 10700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:37.049108 10700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:37.049144 10700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:35:37.049181 10700 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:35:37.049213 10700 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 06:35:37.054661 10700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:41645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:40921
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:40921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577308481727-5208-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:37.055765 10700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:37.056816 10700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:37.065402 10706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:37.065400 10708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:37.065397 10705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:37.065763 10700 server_base.cc:1034] running on GCE node
I20260928 06:35:37.066437 10700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:37.067559 10700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:37.068720 10700 hybrid_clock.cc:648] HybridClock initialized: now 1790577337068706 us; error 27 us; skew 500 ppm
I20260928 06:35:37.070683 10700 webserver.cc:460] Webserver started at http://127.5.22.62:34343/ using document root <none> and password file <none>
I20260928 06:35:37.071246 10700 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:37.071307 10700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:37.071551 10700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:37.073568 10700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "1d2d32ebf24c45d5934cea75155279f7"
format_stamp: "Formatted at 2026-09-28 06:35:37 on dist-test-slave-j0qf"
server_key: "c360f7eff55d838e7227c7132a216e2c"
server_key_iv: "e22f0b671658e36b68b576327d38888f"
server_key_version: "encryptionkey@0"
I20260928 06:35:37.074122 10700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "1d2d32ebf24c45d5934cea75155279f7"
format_stamp: "Formatted at 2026-09-28 06:35:37 on dist-test-slave-j0qf"
server_key: "c360f7eff55d838e7227c7132a216e2c"
server_key_iv: "e22f0b671658e36b68b576327d38888f"
server_key_version: "encryptionkey@0"
I20260928 06:35:37.077804 10700 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:35:37.080336 10714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:37.081625 10700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:37.081763 10700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "1d2d32ebf24c45d5934cea75155279f7"
format_stamp: "Formatted at 2026-09-28 06:35:37 on dist-test-slave-j0qf"
server_key: "c360f7eff55d838e7227c7132a216e2c"
server_key_iv: "e22f0b671658e36b68b576327d38888f"
server_key_version: "encryptionkey@0"
I20260928 06:35:37.081866 10700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:37.102144 10700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:37.102911 10700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:37.103112 10700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:37.109965 10700 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:40921
I20260928 06:35:37.109988 10767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:40921 every 8 connection(s)
I20260928 06:35:37.111027 10700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:37.114172 10768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:37.116796 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 10700
I20260928 06:35:37.116927 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:35:37.117229 5208 external_mini_cluster.cc:1428] Setting key e94addc5df77a9a4580ded39000b4406
I20260928 06:35:37.120491 10768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d2d32ebf24c45d5934cea75155279f7: Bootstrap starting.
I20260928 06:35:37.122709 10768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d2d32ebf24c45d5934cea75155279f7: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:37.123446 10768 log.cc:824] T 00000000000000000000000000000000 P 1d2d32ebf24c45d5934cea75155279f7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:37.125442 10768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d2d32ebf24c45d5934cea75155279f7: No bootstrap required, opened a new log
I20260928 06:35:37.126818 10771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:37.118764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52646 (local address 127.5.22.62:40921)
0928 06:35:37.119430 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:37.119445 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:37.119480 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:35:37.120101 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:37.120118 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:37.120510 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:37.120893 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:37.121400 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.122464 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.123380 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.123912 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.124092 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:37.124974 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:37.124979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:37.124992 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:37.125386 (+ 394us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:37.125446 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:37.125487 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:37.125605 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:37.125787 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 06:35:37.126049 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":142,"threads_started":1}
I20260928 06:35:37.129694 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:37.129599 10768 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d2d32ebf24c45d5934cea75155279f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2d32ebf24c45d5934cea75155279f7" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 40921 } }
I20260928 06:35:37.129881 10768 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d2d32ebf24c45d5934cea75155279f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:37.129967 10768 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d2d32ebf24c45d5934cea75155279f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d2d32ebf24c45d5934cea75155279f7, State: Initialized, Role: FOLLOWER
I20260928 06:35:37.130456 10768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d2d32ebf24c45d5934cea75155279f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2d32ebf24c45d5934cea75155279f7" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 40921 } }
I20260928 06:35:37.130601 10768 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d2d32ebf24c45d5934cea75155279f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:37.130669 10768 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d2d32ebf24c45d5934cea75155279f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:37.130771 10768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d2d32ebf24c45d5934cea75155279f7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:37.131767 10768 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d2d32ebf24c45d5934cea75155279f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2d32ebf24c45d5934cea75155279f7" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 40921 } }
I20260928 06:35:37.132052 10768 leader_election.cc:302] T 00000000000000000000000000000000 P 1d2d32ebf24c45d5934cea75155279f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d2d32ebf24c45d5934cea75155279f7; no voters:
I20260928 06:35:37.132292 10768 leader_election.cc:288] T 00000000000000000000000000000000 P 1d2d32ebf24c45d5934cea75155279f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:37.132411 10773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d2d32ebf24c45d5934cea75155279f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:37.132702 10773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d2d32ebf24c45d5934cea75155279f7 [term 1 LEADER]: Becoming Leader. State: Replica: 1d2d32ebf24c45d5934cea75155279f7, State: Running, Role: LEADER
I20260928 06:35:37.133003 10773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d2d32ebf24c45d5934cea75155279f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2d32ebf24c45d5934cea75155279f7" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 40921 } }
I20260928 06:35:37.133432 10768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d2d32ebf24c45d5934cea75155279f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:37.134603 10774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2d32ebf24c45d5934cea75155279f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d2d32ebf24c45d5934cea75155279f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2d32ebf24c45d5934cea75155279f7" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 40921 } } }
I20260928 06:35:37.134737 10774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2d32ebf24c45d5934cea75155279f7 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:37.134991 10775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2d32ebf24c45d5934cea75155279f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d2d32ebf24c45d5934cea75155279f7. Latest consensus state: current_term: 1 leader_uuid: "1d2d32ebf24c45d5934cea75155279f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2d32ebf24c45d5934cea75155279f7" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 40921 } } }
I20260928 06:35:37.135107 10775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2d32ebf24c45d5934cea75155279f7 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:37.135177 10782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:37.138000 10782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:37.143163 10782 catalog_manager.cc:1347] Generated new cluster ID: b2960efc3520471cb28629bb15929f6a
I20260928 06:35:37.143252 10782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:37.149464 10782 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:37.150565 10782 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:37.157779 10782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d2d32ebf24c45d5934cea75155279f7: Generated new TSK 0
I20260928 06:35:37.158449 10782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:37.632181 10771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:37.168564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52660 (local address 127.5.22.62:40921)
0928 06:35:37.168689 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:37.168693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:37.168783 (+ 90us) server_negotiation.cc:407] Connection header received
0928 06:35:37.168970 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:37.168973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:37.169017 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:37.169121 (+ 104us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:35:37.169354 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.169877 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.170828 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.171066 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.171186 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:37.171492 (+ 306us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:35:37.631789 (+460297us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 06:35:37.632062 (+ 273us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52660: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 06:35:37.632279 10771 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52660: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 06:35:37.632530 10797 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:37.168411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:40921 (local address 127.0.0.1:52660)
0928 06:35:37.168693 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:37.168709 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:37.168839 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:37.169120 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:37.169124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:37.169131 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:35:37.169245 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:37.169251 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.170013 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.170017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:37.170701 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:37.170710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.171185 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.171207 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:37.171295 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:37.171317 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:35:37.632031 (+460714us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 06:35:37.632220 (+ 189us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.22.62:40921: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":108,"threads_started":1}
I20260928 06:35:37.633296 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 10700
2026-09-28T06:35:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (731 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (124 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (126 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:37 dist-test-slave-j0qf krb5kdc[10809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:37 dist-test-slave-j0qf krb5kdc[10809](info): set up 2 sockets
Sep 28 06:35:37 dist-test-slave-j0qf krb5kdc[10809](info): commencing operation
krb5kdc: starting...
W20260928 06:35:38.945254 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:35:39 dist-test-slave-j0qf krb5kdc[10809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:35:39.073366 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:36833
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:36833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:39.183411 10822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:39.183727 10822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:39.183797 10822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:39.187816 10822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:39.187899 10822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:39.187934 10822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:39.187983 10822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:39.188016 10822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:39.193305 10822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:36833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:36833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:39.194478 10822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:39.195575 10822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:39.204154 10830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:39.204164 10827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:39.204149 10828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:39.204506 10822 server_base.cc:1034] running on GCE node
I20260928 06:35:39.205039 10822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:39.206200 10822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:39.207407 10822 hybrid_clock.cc:648] HybridClock initialized: now 1790577339207385 us; error 41 us; skew 500 ppm
Sep 28 06:35:39 dist-test-slave-j0qf krb5kdc[10809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577339, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:39.213640 10822 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:35:39.214565 10822 webserver.cc:460] Webserver started at http://127.5.22.62:33547/ using document root <none> and password file <none>
I20260928 06:35:39.215140 10822 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:39.215231 10822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:39.215461 10822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:39.217485 10822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "8b63288dbe174e84a463148b0d63c730"
format_stamp: "Formatted at 2026-09-28 06:35:39 on dist-test-slave-j0qf"
server_key: "92e97fe085e172a197b7f7155c183c0e"
server_key_iv: "aef5fb8e96be61a3c0d0075aa2eef408"
server_key_version: "encryptionkey@0"
I20260928 06:35:39.218050 10822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "8b63288dbe174e84a463148b0d63c730"
format_stamp: "Formatted at 2026-09-28 06:35:39 on dist-test-slave-j0qf"
server_key: "92e97fe085e172a197b7f7155c183c0e"
server_key_iv: "aef5fb8e96be61a3c0d0075aa2eef408"
server_key_version: "encryptionkey@0"
I20260928 06:35:39.222111 10822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 06:35:39.224470 10837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:39.225652 10822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:39.225781 10822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "8b63288dbe174e84a463148b0d63c730"
format_stamp: "Formatted at 2026-09-28 06:35:39 on dist-test-slave-j0qf"
server_key: "92e97fe085e172a197b7f7155c183c0e"
server_key_iv: "aef5fb8e96be61a3c0d0075aa2eef408"
server_key_version: "encryptionkey@0"
I20260928 06:35:39.225891 10822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:39.239171 10822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:39.242759 10822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:39.242908 10822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:39.250439 10822 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:36833
I20260928 06:35:39.250466 10889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:36833 every 8 connection(s)
I20260928 06:35:39.251381 10822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:39.254112 10890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:39.259668 10890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b63288dbe174e84a463148b0d63c730: Bootstrap starting.
I20260928 06:35:39.259758 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 10822
I20260928 06:35:39.259909 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:35:39.260171 5208 external_mini_cluster.cc:1428] Setting key b8c355caafcb588bbd9ddd3f76321624
I20260928 06:35:39.262456 10890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b63288dbe174e84a463148b0d63c730: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:39.263386 10890 log.cc:824] T 00000000000000000000000000000000 P 8b63288dbe174e84a463148b0d63c730: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:39.265659 10890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b63288dbe174e84a463148b0d63c730: No bootstrap required, opened a new log
I20260928 06:35:39.269908 10890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b63288dbe174e84a463148b0d63c730 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b63288dbe174e84a463148b0d63c730" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 36833 } }
I20260928 06:35:39.270219 10890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b63288dbe174e84a463148b0d63c730 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Sep 28 06:35:39 dist-test-slave-j0qf krb5kdc[10809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:39.270284 10890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b63288dbe174e84a463148b0d63c730 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b63288dbe174e84a463148b0d63c730, State: Initialized, Role: FOLLOWER
I20260928 06:35:39.270792 10890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b63288dbe174e84a463148b0d63c730 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b63288dbe174e84a463148b0d63c730" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 36833 } }
I20260928 06:35:39.270952 10890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b63288dbe174e84a463148b0d63c730 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:39.271023 10890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b63288dbe174e84a463148b0d63c730 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:39.271142 10890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b63288dbe174e84a463148b0d63c730 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:39.272505 10890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b63288dbe174e84a463148b0d63c730 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b63288dbe174e84a463148b0d63c730" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 36833 } }
I20260928 06:35:39.272886 10890 leader_election.cc:302] T 00000000000000000000000000000000 P 8b63288dbe174e84a463148b0d63c730 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b63288dbe174e84a463148b0d63c730; no voters:
I20260928 06:35:39.273327 10890 leader_election.cc:288] T 00000000000000000000000000000000 P 8b63288dbe174e84a463148b0d63c730 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:39.273456 10895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b63288dbe174e84a463148b0d63c730 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:39.273773 10895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b63288dbe174e84a463148b0d63c730 [term 1 LEADER]: Becoming Leader. State: Replica: 8b63288dbe174e84a463148b0d63c730, State: Running, Role: LEADER
I20260928 06:35:39.274150 10895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b63288dbe174e84a463148b0d63c730 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b63288dbe174e84a463148b0d63c730" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 36833 } }
I20260928 06:35:39.274652 10890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b63288dbe174e84a463148b0d63c730 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:39.276079 10896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b63288dbe174e84a463148b0d63c730 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b63288dbe174e84a463148b0d63c730" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b63288dbe174e84a463148b0d63c730" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 36833 } } }
I20260928 06:35:39.276216 10896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b63288dbe174e84a463148b0d63c730 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:39.276439 10896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b63288dbe174e84a463148b0d63c730 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b63288dbe174e84a463148b0d63c730. Latest consensus state: current_term: 1 leader_uuid: "8b63288dbe174e84a463148b0d63c730" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b63288dbe174e84a463148b0d63c730" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 36833 } } }
I20260928 06:35:39.276544 10896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b63288dbe174e84a463148b0d63c730 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:39.277370 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.261605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56004 (local address 127.5.22.62:36833)
0928 06:35:39.262021 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:39.262040 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:39.262081 (+ 41us) server_negotiation.cc:407] Connection header received
0928 06:35:39.263009 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:39.263030 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:39.263478 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:39.263874 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:39.264899 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.266361 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.267218 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.267947 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.268110 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.270970 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:39.270977 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:39.270995 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:39.271033 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:39.273138 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.273764 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.273771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.273778 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.273943 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.274360 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.274364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.274366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.274916 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:39.275145 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:39.275697 (+ 552us) server_negotiation.cc:299] Negotiation successful
0928 06:35:39.276124 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":123,"threads_started":1}
W20260928 06:35:39.279927 10910 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b63288dbe174e84a463148b0d63c730: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:35:39.280023 10910 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:35:39.280083 10907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:39.280732 10907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:39.286484 10907 catalog_manager.cc:1347] Generated new cluster ID: c4b8c16be47844a8837be17e5aab5727
I20260928 06:35:39.286556 10907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:39.300357 10907 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:39.301733 10907 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:39.311787 10907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b63288dbe174e84a463148b0d63c730: Generated new TSK 0
I20260928 06:35:39.312487 10907 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.22.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
I20260928 06:35:39.351660 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:36833
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:39.465298 10916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:39.465569 10916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:39.465628 10916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:39.469853 10916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:39.469913 10916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:39.469978 10916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:35:39.475410 10916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:36833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:39.476564 10916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:39.477675 10916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:39.487327 10921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:39.487396 10922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:39.487357 10924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:39.487623 10916 server_base.cc:1034] running on GCE node
I20260928 06:35:39.488020 10916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:39.488641 10916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:39.489854 10916 hybrid_clock.cc:648] HybridClock initialized: now 1790577339489818 us; error 45 us; skew 500 ppm
Sep 28 06:35:39 dist-test-slave-j0qf krb5kdc[10809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577339, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:39.492637 10916 init.cc:377] Logged in from keytab as kudu/127.5.22.1@KRBTEST.COM (short username kudu)
I20260928 06:35:39.493525 10916 webserver.cc:460] Webserver started at http://127.5.22.1:43469/ using document root <none> and password file <none>
I20260928 06:35:39.494056 10916 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:39.494117 10916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:39.494369 10916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:39.496361 10916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "f85a5d10e81541959cc9d75cbf614740"
format_stamp: "Formatted at 2026-09-28 06:35:39 on dist-test-slave-j0qf"
server_key: "cf5e652626f16a159ff818eea84954ab"
server_key_iv: "391f9286a60b3d1f5776684dd689684c"
server_key_version: "encryptionkey@0"
I20260928 06:35:39.496933 10916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "f85a5d10e81541959cc9d75cbf614740"
format_stamp: "Formatted at 2026-09-28 06:35:39 on dist-test-slave-j0qf"
server_key: "cf5e652626f16a159ff818eea84954ab"
server_key_iv: "391f9286a60b3d1f5776684dd689684c"
server_key_version: "encryptionkey@0"
I20260928 06:35:39.500528 10916 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 06:35:39.502933 10931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:39.504205 10916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:35:39.504318 10916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "f85a5d10e81541959cc9d75cbf614740"
format_stamp: "Formatted at 2026-09-28 06:35:39 on dist-test-slave-j0qf"
server_key: "cf5e652626f16a159ff818eea84954ab"
server_key_iv: "391f9286a60b3d1f5776684dd689684c"
server_key_version: "encryptionkey@0"
I20260928 06:35:39.504418 10916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:39.533083 10916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:39.536022 10916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:39.536190 10916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:39.536789 10916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:39.537643 10916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:39.537690 10916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:39.537755 10916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:39.537787 10916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:39.546861 10916 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:38551
I20260928 06:35:39.546885 11044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:38551 every 8 connection(s)
I20260928 06:35:39.547756 10916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:39.547983 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 10916
I20260928 06:35:39.548125 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:39.548461 5208 external_mini_cluster.cc:1428] Setting key e5744f0c0cdb403fb5d232c482637e81
Sep 28 06:35:39 dist-test-slave-j0qf krb5kdc[10809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577339, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.22.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.2@KRBTEST.COM" created.
I20260928 06:35:39.561451 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.549690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:50837 (local address 127.5.22.62:36833)
0928 06:35:39.549829 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:39.549834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:39.550841 (+ 1007us) server_negotiation.cc:407] Connection header received
0928 06:35:39.552209 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:39.552213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:39.552268 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:39.552360 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:39.553544 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.554079 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.555274 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.555515 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.555633 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.558050 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:39.558053 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:39.558055 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:39.558079 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:39.559405 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.559900 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.559904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.559906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.559987 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.560330 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.560333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.560335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.560562 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:39.560668 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:39.561148 (+ 480us) server_negotiation.cc:299] Negotiation successful
0928 06:35:39.561279 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:35:39.562584 11047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.550187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:36833 (local address 127.5.22.1:50837)
0928 06:35:39.550702 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:39.550746 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:39.552005 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:39.552574 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:39.552582 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:39.552934 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:39.553399 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.553411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.554202 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.554210 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.555148 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.555157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.555671 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.555675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.555841 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.556469 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:39.556485 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:39.557875 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:39.559523 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:39.559529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:39.559552 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:39.559798 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:39.560088 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:39.560092 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:39.560094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:39.560235 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:39.560666 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:39.560673 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:39.560918 (+ 245us) client_negotiation.cc:769] Sending connection context
0928 06:35:39.561105 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 06:35:39.561423 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":171,"threads_started":1}
I20260928 06:35:39.563822 11045 heartbeater.cc:347] Connected to a master server at 127.5.22.62:36833
I20260928 06:35:39.564081 11045 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:39.564570 11045 heartbeater.cc:511] Master 127.5.22.62:36833 requested a full tablet report, sending...
I20260928 06:35:39.566056 10854 ts_manager.cc:194] Registered new tserver with Master: f85a5d10e81541959cc9d75cbf614740 (127.5.22.1:38551)
I20260928 06:35:39.567231 10854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.1@KRBTEST.COM'} at 127.5.22.1:50837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
I20260928 06:35:39.575798 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:36833
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:39.684947 11050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:39.685196 11050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:39.685238 11050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:39.689198 11050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:39.689253 11050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:39.689309 11050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:35:39.694446 11050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:36833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:39.695369 11050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:39.696292 11050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:39.705673 11058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:39.705649 11055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:39.705632 11056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:39.705962 11050 server_base.cc:1034] running on GCE node
I20260928 06:35:39.706559 11050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:39.707108 11050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:39.708287 11050 hybrid_clock.cc:648] HybridClock initialized: now 1790577339708255 us; error 50 us; skew 500 ppm
Sep 28 06:35:39 dist-test-slave-j0qf krb5kdc[10809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577339, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:39.711047 11050 init.cc:377] Logged in from keytab as kudu/127.5.22.2@KRBTEST.COM (short username kudu)
I20260928 06:35:39.711920 11050 webserver.cc:460] Webserver started at http://127.5.22.2:38643/ using document root <none> and password file <none>
I20260928 06:35:39.712471 11050 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:39.712559 11050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:39.712772 11050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:39.714787 11050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "ea5fad18024e4d21b3664a5fabfda87e"
format_stamp: "Formatted at 2026-09-28 06:35:39 on dist-test-slave-j0qf"
server_key: "83e791a108fbb5bdda81b0840f803705"
server_key_iv: "d4256b82ada016050b13d6dee9d92f99"
server_key_version: "encryptionkey@0"
I20260928 06:35:39.715337 11050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "ea5fad18024e4d21b3664a5fabfda87e"
format_stamp: "Formatted at 2026-09-28 06:35:39 on dist-test-slave-j0qf"
server_key: "83e791a108fbb5bdda81b0840f803705"
server_key_iv: "d4256b82ada016050b13d6dee9d92f99"
server_key_version: "encryptionkey@0"
I20260928 06:35:39.718922 11050 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:39.721197 11065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:39.722445 11050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:35:39.722568 11050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "ea5fad18024e4d21b3664a5fabfda87e"
format_stamp: "Formatted at 2026-09-28 06:35:39 on dist-test-slave-j0qf"
server_key: "83e791a108fbb5bdda81b0840f803705"
server_key_iv: "d4256b82ada016050b13d6dee9d92f99"
server_key_version: "encryptionkey@0"
I20260928 06:35:39.722667 11050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:39.733305 11050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:39.736460 11050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:39.736651 11050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:39.737206 11050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:39.738018 11050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:39.738080 11050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:39.738147 11050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:39.738195 11050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:39.747079 11050 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:36399
I20260928 06:35:39.747112 11178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:36399 every 8 connection(s)
I20260928 06:35:39.747944 11050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:39.752195 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 11050
I20260928 06:35:39.752311 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:39.752599 5208 external_mini_cluster.cc:1428] Setting key a9cdbb8b22d19f97f0ab9aae25aa1d2f
Sep 28 06:35:39 dist-test-slave-j0qf krb5kdc[10809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577339, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.22.3@KRBTEST.COM; defaulting to no policy
I20260928 06:35:39.762444 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.749708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:48691 (local address 127.5.22.62:36833)
0928 06:35:39.749849 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:39.749853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:39.750721 (+ 868us) server_negotiation.cc:407] Connection header received
0928 06:35:39.752030 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:39.752034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:39.752096 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:39.752201 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:39.753529 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.754196 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.756019 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.756422 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.756544 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.758925 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:39.758929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:39.758931 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:39.758961 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:39.760438 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.760930 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.760934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.760936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.761021 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.761354 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.761357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.761359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.761537 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:39.761638 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:39.762124 (+ 486us) server_negotiation.cc:299] Negotiation successful
0928 06:35:39.762243 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 06:35:39.763331 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.750056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:36833 (local address 127.5.22.2:48691)
0928 06:35:39.750566 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:39.750610 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:39.751824 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:39.752392 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:39.752402 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:39.752827 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:39.753399 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.753413 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.754358 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.754362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.755883 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.755899 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.756592 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.756594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.756730 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.757315 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:39.757331 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:39.758765 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:39.760555 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:39.760560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:39.760575 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:39.760824 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:39.761115 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:39.761118 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:39.761120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:39.761254 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:39.761636 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:39.761642 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:39.761886 (+ 244us) client_negotiation.cc:769] Sending connection context
0928 06:35:39.762068 (+ 182us) client_negotiation.cc:240] Negotiation successful
0928 06:35:39.762410 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":118,"threads_started":1}
I20260928 06:35:39.764474 11179 heartbeater.cc:347] Connected to a master server at 127.5.22.62:36833
I20260928 06:35:39.764714 11179 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.3@KRBTEST.COM" created.
I20260928 06:35:39.765292 11179 heartbeater.cc:511] Master 127.5.22.62:36833 requested a full tablet report, sending...
I20260928 06:35:39.766430 10854 ts_manager.cc:194] Registered new tserver with Master: ea5fad18024e4d21b3664a5fabfda87e (127.5.22.2:36399)
I20260928 06:35:39.766999 10854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.2@KRBTEST.COM'} at 127.5.22.2:48691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
I20260928 06:35:39.781069 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:36833
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:39.894847 11184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:39.895126 11184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:39.895203 11184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:39.899281 11184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:39.899340 11184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:39.899439 11184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:35:39.904901 11184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:36833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:39.906066 11184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:39.907320 11184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:39.917454 11192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:39.917528 11189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:39.917482 11190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:39.917745 11184 server_base.cc:1034] running on GCE node
I20260928 06:35:39.918291 11184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:39.919020 11184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:39.920193 11184 hybrid_clock.cc:648] HybridClock initialized: now 1790577339920176 us; error 29 us; skew 500 ppm
Sep 28 06:35:39 dist-test-slave-j0qf krb5kdc[10809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577339, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:39.923033 11184 init.cc:377] Logged in from keytab as kudu/127.5.22.3@KRBTEST.COM (short username kudu)
I20260928 06:35:39.923926 11184 webserver.cc:460] Webserver started at http://127.5.22.3:44565/ using document root <none> and password file <none>
I20260928 06:35:39.924490 11184 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:39.924580 11184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:39.924804 11184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:39.927172 11184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "c0c0aeceb06249929c4d99e72b9dc8c9"
format_stamp: "Formatted at 2026-09-28 06:35:39 on dist-test-slave-j0qf"
server_key: "335db5d2bbc963a374b69cda2ba120c4"
server_key_iv: "bd261f8f36e52f424bcfa9aecad3f771"
server_key_version: "encryptionkey@0"
I20260928 06:35:39.927735 11184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "c0c0aeceb06249929c4d99e72b9dc8c9"
format_stamp: "Formatted at 2026-09-28 06:35:39 on dist-test-slave-j0qf"
server_key: "335db5d2bbc963a374b69cda2ba120c4"
server_key_iv: "bd261f8f36e52f424bcfa9aecad3f771"
server_key_version: "encryptionkey@0"
I20260928 06:35:39.931725 11184 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 06:35:39.934224 11199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:39.935506 11184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 06:35:39.935634 11184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "c0c0aeceb06249929c4d99e72b9dc8c9"
format_stamp: "Formatted at 2026-09-28 06:35:39 on dist-test-slave-j0qf"
server_key: "335db5d2bbc963a374b69cda2ba120c4"
server_key_iv: "bd261f8f36e52f424bcfa9aecad3f771"
server_key_version: "encryptionkey@0"
I20260928 06:35:39.935745 11184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:39.947033 11184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:39.949915 11184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:39.950127 11184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:39.950778 11184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:39.951606 11184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:39.951673 11184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:39.951745 11184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:39.951785 11184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:39.960685 11184 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:35935
I20260928 06:35:39.960706 11312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:35935 every 8 connection(s)
I20260928 06:35:39.961594 11184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:39.967769 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 11184
I20260928 06:35:39.967883 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:39.968168 5208 external_mini_cluster.cc:1428] Setting key 19779ff891e349895e9cb6f0018b0aee
Sep 28 06:35:39 dist-test-slave-j0qf krb5kdc[10809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577339, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:39.975867 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.963602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:33671 (local address 127.5.22.62:36833)
0928 06:35:39.963752 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:39.963756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:39.964539 (+ 783us) server_negotiation.cc:407] Connection header received
0928 06:35:39.965945 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:39.965948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:39.966009 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:39.966119 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:39.967420 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.968234 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.969635 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.969886 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.970004 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.972574 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:39.972578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:39.972580 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:39.972603 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:39.973790 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.974304 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.974351 (+ 47us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.974353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.974435 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.974763 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.974768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.974769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.974948 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:39.975055 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:39.975584 (+ 529us) server_negotiation.cc:299] Negotiation successful
0928 06:35:39.975705 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 06:35:39.976734 11315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.963921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:36833 (local address 127.5.22.3:33671)
0928 06:35:39.964391 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:39.964433 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:39.965727 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:39.966291 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:39.966297 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:39.966768 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:39.967266 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.967279 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.968361 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.968364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.969468 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.969477 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.970055 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.970059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.970272 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.970935 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:39.970951 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:39.972388 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:39.973907 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:39.973913 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:39.973927 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:39.974185 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:39.974524 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:39.974528 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:39.974530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:39.974668 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:39.975062 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:39.975070 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:39.975344 (+ 274us) client_negotiation.cc:769] Sending connection context
0928 06:35:39.975528 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 06:35:39.975869 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":119,"threads_started":1}
I20260928 06:35:39.977917 11313 heartbeater.cc:347] Connected to a master server at 127.5.22.62:36833
I20260928 06:35:39.978170 11313 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:39.978703 11313 heartbeater.cc:511] Master 127.5.22.62:36833 requested a full tablet report, sending...
I20260928 06:35:39.979735 10854 ts_manager.cc:194] Registered new tserver with Master: c0c0aeceb06249929c4d99e72b9dc8c9 (127.5.22.3:35935)
I20260928 06:35:39.980314 10854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.3@KRBTEST.COM'} at 127.5.22.3:33671
I20260928 06:35:39.982306 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/test-admin.keytab.
Sep 28 06:35:39 dist-test-slave-j0qf krb5kdc[10809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:39.999681 5208 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 06:35:40.002729 11320 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:40.001095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42520 (local address 127.5.22.1:38551)
0928 06:35:40.001484 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:40.001498 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:40.001579 (+ 81us) server_negotiation.cc:407] Connection header received
0928 06:35:40.001633 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:40.001639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:40.001834 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:40.001963 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:40.002505 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42520: BlockingRecv error: recv got EOF from 127.0.0.1:42520 (error 108)
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":132,"threads_started":1}
W20260928 06:35:40.002758 5208 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/krb5cc' of type FILE
Sep 28 06:35:40 dist-test-slave-j0qf krb5kdc[10809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:40.004141 5208 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 06:35:40 dist-test-slave-j0qf krb5kdc[10809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:40.005097 5208 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 06:35:40 dist-test-slave-j0qf krb5kdc[10809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.1@KRBTEST.COM
I20260928 06:35:40.015344 11320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:40.005960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42526 (local address 127.5.22.1:38551)
0928 06:35:40.006089 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:40.006093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:40.006188 (+ 95us) server_negotiation.cc:407] Connection header received
0928 06:35:40.006284 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:40.006287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:40.006418 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:40.006487 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:40.007118 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:40.008071 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:40.009009 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:40.009445 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:40.009539 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:40.011864 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:40.011873 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:40.011886 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:40.011910 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:40.013555 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:40.013976 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:40.013981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:40.013982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:40.014077 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:40.014392 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:40.014395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:40.014396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:40.014639 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:40.014817 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:40.014963 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 06:35:40.015166 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 06:35:40.019237 11320 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:40.018121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42530 (local address 127.5.22.1:38551)
0928 06:35:40.018249 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:40.018253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:40.018431 (+ 178us) server_negotiation.cc:407] Connection header received
0928 06:35:40.018556 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:40.018559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:40.018599 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:40.018668 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:40.019116 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42530: BlockingRecv error: recv got EOF from 127.0.0.1:42530 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 06:35:40.019518 5208 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/krb5cc)
Sep 28 06:35:40 dist-test-slave-j0qf krb5kdc[10809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:40.020807 5208 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 06:35:40 dist-test-slave-j0qf krb5kdc[10809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:40.021757 5208 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 06:35:40 dist-test-slave-j0qf krb5kdc[10809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.1@KRBTEST.COM
I20260928 06:35:40.030537 11320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:40.022644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42536 (local address 127.5.22.1:38551)
0928 06:35:40.022787 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:40.022791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:40.022835 (+ 44us) server_negotiation.cc:407] Connection header received
0928 06:35:40.022928 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:40.022930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:40.022965 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:40.023037 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:40.023496 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:40.024068 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:40.024823 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:40.025104 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:40.025198 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:40.027486 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:40.027489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:40.027491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:40.027521 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:40.029124 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:40.029473 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:40.029475 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:40.029477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:40.029547 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:40.029727 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:40.029729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:40.029730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:40.029868 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:40.029957 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:40.030216 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 06:35:40.030367 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 06:35:40.032979 11320 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:40.032143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42550 (local address 127.5.22.1:38551)
0928 06:35:40.032265 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:40.032268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:40.032294 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:35:40.032406 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:40.032408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:40.032441 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:40.032502 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:40.032871 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42550: BlockingRecv error: recv got EOF from 127.0.0.1:42550 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 06:35:40.033046 5208 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577308481727-5208-0/krb5kdc/krb5cc' of type FILE
Sep 28 06:35:40 dist-test-slave-j0qf krb5kdc[10809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:40.034046 5208 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 06:35:40 dist-test-slave-j0qf krb5kdc[10809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:40.034921 5208 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 06:35:40 dist-test-slave-j0qf krb5kdc[10809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.1@KRBTEST.COM
I20260928 06:35:40.043138 11320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:40.035760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42556 (local address 127.5.22.1:38551)
0928 06:35:40.035877 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:40.035881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:40.035978 (+ 97us) server_negotiation.cc:407] Connection header received
0928 06:35:40.036091 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:40.036094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:40.036134 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:40.036212 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:40.036697 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:40.037180 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:40.037952 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:40.038167 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:40.038270 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:40.040550 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:40.040556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:40.040560 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:40.040586 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:40.041744 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:40.042109 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:40.042111 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:40.042112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:40.042177 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:40.042439 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:40.042441 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:40.042442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:40.042587 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:40.042686 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:40.042862 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 06:35:40.042967 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:35:40.044081 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 10916
I20260928 06:35:40.050426 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 11050
I20260928 06:35:40.055577 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 11184
I20260928 06:35:40.060931 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 10822
2026-09-28T06:35:40Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2178 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:40 dist-test-slave-j0qf krb5kdc[11334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:40 dist-test-slave-j0qf krb5kdc[11334](info): set up 2 sockets
Sep 28 06:35:40 dist-test-slave-j0qf krb5kdc[11334](info): commencing operation
krb5kdc: starting...
W20260928 06:35:41.141687 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:35:41 dist-test-slave-j0qf krb5kdc[11334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.62.keytab.
Entry for principal oryx/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.62.keytab.
I20260928 06:35:41.266091 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:38463
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.62.keytab
--principal=oryx/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:38463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:41.373662 11347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:41.373916 11347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:41.373955 11347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:41.377918 11347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:41.377978 11347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:41.377997 11347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:41.378013 11347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:41.378074 11347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:41.383248 11347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:38463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.62.keytab
--principal=oryx/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:38463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:41.384274 11347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:41.385284 11347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:41.393678 11353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:41.393713 11352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:41.393791 11347 server_base.cc:1034] running on GCE node
W20260928 06:35:41.393705 11355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:41.394484 11347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:41.395589 11347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:41.396806 11347 hybrid_clock.cc:648] HybridClock initialized: now 1790577341396791 us; error 27 us; skew 500 ppm
Sep 28 06:35:41 dist-test-slave-j0qf krb5kdc[11334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577341, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:41.399510 11347 init.cc:377] Logged in from keytab as oryx/127.5.22.62@KRBTEST.COM (short username oryx)
I20260928 06:35:41.400350 11347 webserver.cc:460] Webserver started at http://127.5.22.62:37153/ using document root <none> and password file <none>
I20260928 06:35:41.400906 11347 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:41.400996 11347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:41.401278 11347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:41.403609 11347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "5973a583576742e5a79e1b84b9e53783"
format_stamp: "Formatted at 2026-09-28 06:35:41 on dist-test-slave-j0qf"
server_key: "a08aa14fa4f671250b60c2f17c6ba874"
server_key_iv: "f5ab141cb0c254f521e56559fc167d83"
server_key_version: "encryptionkey@0"
I20260928 06:35:41.404175 11347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "5973a583576742e5a79e1b84b9e53783"
format_stamp: "Formatted at 2026-09-28 06:35:41 on dist-test-slave-j0qf"
server_key: "a08aa14fa4f671250b60c2f17c6ba874"
server_key_iv: "f5ab141cb0c254f521e56559fc167d83"
server_key_version: "encryptionkey@0"
I20260928 06:35:41.407923 11347 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260928 06:35:41.410229 11362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:41.411515 11347 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:41.411614 11347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "5973a583576742e5a79e1b84b9e53783"
format_stamp: "Formatted at 2026-09-28 06:35:41 on dist-test-slave-j0qf"
server_key: "a08aa14fa4f671250b60c2f17c6ba874"
server_key_iv: "f5ab141cb0c254f521e56559fc167d83"
server_key_version: "encryptionkey@0"
I20260928 06:35:41.411742 11347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:41.434564 11347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:41.437659 11347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:41.437860 11347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:41.445019 11347 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:38463
I20260928 06:35:41.445307 11414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:38463 every 8 connection(s)
I20260928 06:35:41.446007 11347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:41.448794 11415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:41.452246 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 11347
I20260928 06:35:41.452402 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:35:41.452713 5208 external_mini_cluster.cc:1428] Setting key 8aa08b658edc5b0f214ae8db5641825e
I20260928 06:35:41.454877 11415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783: Bootstrap starting.
I20260928 06:35:41.457082 11415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:41.457793 11415 log.cc:824] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:41.459647 11415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783: No bootstrap required, opened a new log
Sep 28 06:35:41 dist-test-slave-j0qf krb5kdc[11334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM
I20260928 06:35:41.463321 11415 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } }
I20260928 06:35:41.463558 11415 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:41.463644 11415 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5973a583576742e5a79e1b84b9e53783, State: Initialized, Role: FOLLOWER
I20260928 06:35:41.464095 11415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } }
I20260928 06:35:41.464238 11415 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:41.464318 11415 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:41.464414 11415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:41.465533 11415 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } }
I20260928 06:35:41.465876 11415 leader_election.cc:302] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5973a583576742e5a79e1b84b9e53783; no voters:
I20260928 06:35:41.466192 11415 leader_election.cc:288] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:41.466305 11420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:41.466578 11420 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [term 1 LEADER]: Becoming Leader. State: Replica: 5973a583576742e5a79e1b84b9e53783, State: Running, Role: LEADER
I20260928 06:35:41.466881 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.454128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48176 (local address 127.5.22.62:38463)
0928 06:35:41.454691 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:41.454706 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:41.454744 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:35:41.455409 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:41.455427 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:41.455762 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:41.456096 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:41.456925 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.458007 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.458916 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.459383 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.459575 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.461972 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:41.461977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:41.461990 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:41.462016 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:41.463766 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:41.464195 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:41.464201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:41.464206 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:41.464344 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:41.464689 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:41.464694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:41.464696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:41.465179 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:41.465384 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:41.465584 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 06:35:41.465878 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":145,"threads_started":1}
I20260928 06:35:41.466921 11420 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } }
I20260928 06:35:41.467422 11415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:41.468667 11421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5973a583576742e5a79e1b84b9e53783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } }
I20260928 06:35:41.468715 11422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5973a583576742e5a79e1b84b9e53783. Latest consensus state: current_term: 1 leader_uuid: "5973a583576742e5a79e1b84b9e53783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } }
I20260928 06:35:41.468782 11421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:41.468807 11422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:41.469267 11424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:41.472347 11424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:41.477371 11424 catalog_manager.cc:1347] Generated new cluster ID: 6c9264db8f9d43f884348bed61629dcc
I20260928 06:35:41.477442 11424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:41.484282 11424 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:41.485446 11424 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:41.490882 11424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783: Generated new TSK 0
I20260928 06:35:41.491648 11424 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.22.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.22.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.22.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.1.keytab.
Entry for principal oryx/127.5.22.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.1.keytab.
I20260928 06:35:41.527854 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:38463
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.1.keytab
--principal=oryx/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:41.638029 11441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:41.638307 11441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:41.638453 11441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:41.642494 11441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:41.642551 11441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:41.642645 11441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:35:41.648345 11441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.1.keytab
--principal=oryx/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:38463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:41.649399 11441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:41.650477 11441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:41.659781 11449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:41.659929 11447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:41.659824 11446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:41.660054 11441 server_base.cc:1034] running on GCE node
I20260928 06:35:41.660620 11441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:41.661235 11441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:41.662382 11441 hybrid_clock.cc:648] HybridClock initialized: now 1790577341662373 us; error 44 us; skew 500 ppm
Sep 28 06:35:41 dist-test-slave-j0qf krb5kdc[11334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577341, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:41.665319 11441 init.cc:377] Logged in from keytab as oryx/127.5.22.1@KRBTEST.COM (short username oryx)
I20260928 06:35:41.666087 11441 webserver.cc:460] Webserver started at http://127.5.22.1:39681/ using document root <none> and password file <none>
I20260928 06:35:41.666668 11441 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:41.666730 11441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:41.666889 11441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:41.668843 11441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "50444827a7fb4637bee08e5afafa8f84"
format_stamp: "Formatted at 2026-09-28 06:35:41 on dist-test-slave-j0qf"
server_key: "eaef8d1e576462505b725e1b172288d2"
server_key_iv: "f66b967acdd970dace87d8791946b495"
server_key_version: "encryptionkey@0"
I20260928 06:35:41.669340 11441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "50444827a7fb4637bee08e5afafa8f84"
format_stamp: "Formatted at 2026-09-28 06:35:41 on dist-test-slave-j0qf"
server_key: "eaef8d1e576462505b725e1b172288d2"
server_key_iv: "f66b967acdd970dace87d8791946b495"
server_key_version: "encryptionkey@0"
I20260928 06:35:41.673091 11441 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260928 06:35:41.675444 11456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:41.676657 11441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:35:41.676781 11441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "50444827a7fb4637bee08e5afafa8f84"
format_stamp: "Formatted at 2026-09-28 06:35:41 on dist-test-slave-j0qf"
server_key: "eaef8d1e576462505b725e1b172288d2"
server_key_iv: "f66b967acdd970dace87d8791946b495"
server_key_version: "encryptionkey@0"
I20260928 06:35:41.676882 11441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:41.690878 11441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:41.693652 11441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:41.693835 11441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:41.694449 11441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:41.695245 11441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:41.695291 11441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:41.695354 11441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:41.695384 11441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:41.703662 11441 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:44565
I20260928 06:35:41.703689 11569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:44565 every 8 connection(s)
I20260928 06:35:41.704509 11441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:35:41 dist-test-slave-j0qf krb5kdc[11334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577341, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM
I20260928 06:35:41.714238 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 11441
I20260928 06:35:41.714382 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:41.714721 5208 external_mini_cluster.cc:1428] Setting key c0c5a7347d4e487a715874313d08a2f8
I20260928 06:35:41.717840 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.706196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:41613 (local address 127.5.22.62:38463)
0928 06:35:41.706360 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:41.706365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:41.707511 (+ 1146us) server_negotiation.cc:407] Connection header received
0928 06:35:41.708377 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:41.708380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:41.708427 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:41.708511 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:41.709617 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.710151 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.711361 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.711659 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.711748 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.714282 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:41.714286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:41.714288 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:41.714363 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:41.715894 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:41.716380 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:41.716384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:41.716386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:41.716469 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:41.716789 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:41.716794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:41.716796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:41.716974 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:41.717075 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:41.717552 (+ 477us) server_negotiation.cc:299] Negotiation successful
0928 06:35:41.717667 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:35:41.718662 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.706721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.5.22.1:41613)
0928 06:35:41.707365 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:41.707404 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:41.708172 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:41.708681 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:41.708687 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:41.709027 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:41.709488 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.709501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.710253 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.710256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.711246 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.711255 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.711792 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.711795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.711926 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.712537 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:41.712553 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:41.714122 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:41.716010 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:41.716016 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:41.716029 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:41.716277 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:41.716557 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:41.716561 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:41.716563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:41.716698 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:41.717076 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:41.717083 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:41.717315 (+ 232us) client_negotiation.cc:769] Sending connection context
0928 06:35:41.717495 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 06:35:41.717798 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":112,"threads_started":1}
I20260928 06:35:41.719869 11570 heartbeater.cc:347] Connected to a master server at 127.5.22.62:38463
I20260928 06:35:41.720137 11570 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:41.720700 11570 heartbeater.cc:511] Master 127.5.22.62:38463 requested a full tablet report, sending...
I20260928 06:35:41.722443 11376 ts_manager.cc:194] Registered new tserver with Master: 50444827a7fb4637bee08e5afafa8f84 (127.5.22.1:44565)
I20260928 06:35:41.724112 11376 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.22.1@KRBTEST.COM'} at 127.5.22.1:41613
WARNING: no policy specified for oryx/127.5.22.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.22.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.22.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.2.keytab.
Entry for principal oryx/127.5.22.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.2.keytab.
I20260928 06:35:41.741984 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:38463
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.2.keytab
--principal=oryx/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:41.853579 11575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:41.853870 11575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:41.853914 11575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:41.858297 11575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:41.858414 11575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:41.858511 11575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:35:41.864194 11575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.2.keytab
--principal=oryx/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:38463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:41.865239 11575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:41.866282 11575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:41.876190 11580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:41.876145 11581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:41.876153 11583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:41.876456 11575 server_base.cc:1034] running on GCE node
I20260928 06:35:41.876931 11575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:41.877513 11575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:41.878705 11575 hybrid_clock.cc:648] HybridClock initialized: now 1790577341878682 us; error 33 us; skew 500 ppm
Sep 28 06:35:41 dist-test-slave-j0qf krb5kdc[11334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577341, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:41.881834 11575 init.cc:377] Logged in from keytab as oryx/127.5.22.2@KRBTEST.COM (short username oryx)
I20260928 06:35:41.882799 11575 webserver.cc:460] Webserver started at http://127.5.22.2:38429/ using document root <none> and password file <none>
I20260928 06:35:41.883365 11575 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:41.883453 11575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:41.883683 11575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:41.885735 11575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "b63caecbb37e43f58339a2f1a616de2c"
format_stamp: "Formatted at 2026-09-28 06:35:41 on dist-test-slave-j0qf"
server_key: "a0e055edc6a82e03b0f41313470766a5"
server_key_iv: "cd9d931c4c70c38ecb9404ab35fd34f1"
server_key_version: "encryptionkey@0"
I20260928 06:35:41.886366 11575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "b63caecbb37e43f58339a2f1a616de2c"
format_stamp: "Formatted at 2026-09-28 06:35:41 on dist-test-slave-j0qf"
server_key: "a0e055edc6a82e03b0f41313470766a5"
server_key_iv: "cd9d931c4c70c38ecb9404ab35fd34f1"
server_key_version: "encryptionkey@0"
I20260928 06:35:41.890163 11575 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 06:35:41.892673 11590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:41.893891 11575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:41.894024 11575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "b63caecbb37e43f58339a2f1a616de2c"
format_stamp: "Formatted at 2026-09-28 06:35:41 on dist-test-slave-j0qf"
server_key: "a0e055edc6a82e03b0f41313470766a5"
server_key_iv: "cd9d931c4c70c38ecb9404ab35fd34f1"
server_key_version: "encryptionkey@0"
I20260928 06:35:41.894140 11575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:41.911389 11575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:41.914829 11575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:41.915038 11575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:41.915737 11575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:41.916685 11575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:41.916754 11575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:41.916814 11575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:41.916862 11575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:41.925853 11575 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:41085
I20260928 06:35:41.925871 11703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:41085 every 8 connection(s)
I20260928 06:35:41.926831 11575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:41.929011 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 11575
I20260928 06:35:41.929155 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:41.929543 5208 external_mini_cluster.cc:1428] Setting key 8aca7fc7ec8204299ade39396d2d4c8f
Sep 28 06:35:41 dist-test-slave-j0qf krb5kdc[11334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577341, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.5.22.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.22.3@KRBTEST.COM" created.
I20260928 06:35:41.943014 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.928653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:49671 (local address 127.5.22.62:38463)
0928 06:35:41.928796 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:41.928800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:41.930511 (+ 1711us) server_negotiation.cc:407] Connection header received
0928 06:35:41.931623 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:41.931629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:41.931729 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:41.931819 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:41.933218 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.933745 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.935613 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.935844 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.935933 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.938972 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:41.938975 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:41.938978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:41.939004 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:41.940597 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:41.941165 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:41.941169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:41.941171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:41.941255 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:41.941687 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:41.941694 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:41.941698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:41.941913 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:41.942022 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:41.942692 (+ 670us) server_negotiation.cc:299] Negotiation successful
0928 06:35:41.942825 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:35:41.943873 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.929132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.5.22.2:49671)
0928 06:35:41.930284 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:41.930394 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:41.931392 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:41.931972 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:41.931980 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:41.932371 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:41.933027 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.933049 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.933877 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.933880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.935444 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.935459 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.936011 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.936016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.936266 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.936936 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:41.936964 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:41.938782 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:41.940753 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:41.940759 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:41.940776 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:41.941052 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:41.941347 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:41.941360 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:41.941362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:41.941520 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:41.942023 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:41.942031 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:41.942413 (+ 382us) client_negotiation.cc:769] Sending connection context
0928 06:35:41.942632 (+ 219us) client_negotiation.cc:240] Negotiation successful
0928 06:35:41.942982 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"thread_start_us":719,"threads_started":1}
I20260928 06:35:41.945109 11704 heartbeater.cc:347] Connected to a master server at 127.5.22.62:38463
I20260928 06:35:41.945369 11704 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:41.945889 11704 heartbeater.cc:511] Master 127.5.22.62:38463 requested a full tablet report, sending...
I20260928 06:35:41.947014 11376 ts_manager.cc:194] Registered new tserver with Master: b63caecbb37e43f58339a2f1a616de2c (127.5.22.2:41085)
I20260928 06:35:41.947937 11376 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.22.2@KRBTEST.COM'} at 127.5.22.2:49671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.22.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.3.keytab.
Entry for principal oryx/127.5.22.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.3.keytab.
I20260928 06:35:41.959384 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:38463
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.3.keytab
--principal=oryx/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:42.077504 11709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:42.077747 11709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:42.077813 11709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:42.081940 11709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:42.081995 11709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:42.082077 11709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:35:42.087301 11709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.3.keytab
--principal=oryx/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:38463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:42.088285 11709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:42.089248 11709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:42.098619 11715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:42.098619 11717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:42.098773 11709 server_base.cc:1034] running on GCE node
W20260928 06:35:42.098656 11714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:42.099289 11709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:42.099882 11709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:42.101078 11709 hybrid_clock.cc:648] HybridClock initialized: now 1790577342101058 us; error 32 us; skew 500 ppm
Sep 28 06:35:42 dist-test-slave-j0qf krb5kdc[11334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577342, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:42.103744 11709 init.cc:377] Logged in from keytab as oryx/127.5.22.3@KRBTEST.COM (short username oryx)
I20260928 06:35:42.104573 11709 webserver.cc:460] Webserver started at http://127.5.22.3:45357/ using document root <none> and password file <none>
I20260928 06:35:42.105118 11709 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:42.105206 11709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:42.105427 11709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:42.107461 11709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "af81bb4b529c4a58824980974cb7362e"
format_stamp: "Formatted at 2026-09-28 06:35:42 on dist-test-slave-j0qf"
server_key: "6fff47fb134f2585aaa9cd0de7ee8f5b"
server_key_iv: "deba341bacc80f96857f71d0fbb625c9"
server_key_version: "encryptionkey@0"
I20260928 06:35:42.108013 11709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "af81bb4b529c4a58824980974cb7362e"
format_stamp: "Formatted at 2026-09-28 06:35:42 on dist-test-slave-j0qf"
server_key: "6fff47fb134f2585aaa9cd0de7ee8f5b"
server_key_iv: "deba341bacc80f96857f71d0fbb625c9"
server_key_version: "encryptionkey@0"
I20260928 06:35:42.111675 11709 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:35:42.114037 11724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:42.115423 11709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:42.115561 11709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "af81bb4b529c4a58824980974cb7362e"
format_stamp: "Formatted at 2026-09-28 06:35:42 on dist-test-slave-j0qf"
server_key: "6fff47fb134f2585aaa9cd0de7ee8f5b"
server_key_iv: "deba341bacc80f96857f71d0fbb625c9"
server_key_version: "encryptionkey@0"
I20260928 06:35:42.115674 11709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:42.135751 11709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:42.138724 11709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:42.138922 11709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:42.139523 11709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:42.140370 11709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:42.140447 11709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:42.140537 11709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:42.140583 11709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:42.149339 11709 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:37615
I20260928 06:35:42.149365 11837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:37615 every 8 connection(s)
I20260928 06:35:42.150215 11709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:42.155773 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 11709
I20260928 06:35:42.155875 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:42.156140 5208 external_mini_cluster.cc:1428] Setting key 45d56dd139650faf8083e727cdc4a571
Sep 28 06:35:42 dist-test-slave-j0qf krb5kdc[11334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577342, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM
I20260928 06:35:42.163864 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.152033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:42651 (local address 127.5.22.62:38463)
0928 06:35:42.152189 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.152193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.153176 (+ 983us) server_negotiation.cc:407] Connection header received
0928 06:35:42.154110 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.154113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.154178 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.154269 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:42.155444 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.156236 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.157602 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.157905 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.158043 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.160416 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:42.160420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:42.160422 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:42.160447 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:42.162077 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.162604 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.162607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.162609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.162690 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.162963 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.162965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.162967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.163131 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:42.163206 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.163637 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.163735 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 06:35:42.164724 11840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.152390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.5.22.3:42651)
0928 06:35:42.153028 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:42.153068 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:42.153894 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:42.154459 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:42.154465 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:42.154815 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:42.155303 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.155316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.156357 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.156359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.157500 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.157510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.158074 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.158081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.158277 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.158833 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:42.158848 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:42.160244 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:42.162181 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.162185 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.162196 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.162508 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.162772 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.162774 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.162775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.162901 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.163211 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:42.163216 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:42.163424 (+ 208us) client_negotiation.cc:769] Sending connection context
0928 06:35:42.163590 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 06:35:42.163824 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":113,"threads_started":1}
I20260928 06:35:42.165879 11838 heartbeater.cc:347] Connected to a master server at 127.5.22.62:38463
I20260928 06:35:42.166090 11838 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:42.166596 11838 heartbeater.cc:511] Master 127.5.22.62:38463 requested a full tablet report, sending...
I20260928 06:35:42.167696 11376 ts_manager.cc:194] Registered new tserver with Master: af81bb4b529c4a58824980974cb7362e (127.5.22.3:37615)
I20260928 06:35:42.168318 11376 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.22.3@KRBTEST.COM'} at 127.5.22.3:42651
I20260928 06:35:42.169948 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.22.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.22.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.22.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.61.keytab.
Entry for principal oryx/127.5.22.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.61.keytab.
I20260928 06:35:42.197851 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.61:39427
--webserver_interface=127.5.22.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.61.keytab
--principal=oryx/127.5.22.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:38463,127.5.22.61:39427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:42.307860 11843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:42.308127 11843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:42.308195 11843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:42.312225 11843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:42.312278 11843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:42.312296 11843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:42.312330 11843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:42.312372 11843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:42.317662 11843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:38463,127.5.22.61:39427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.61.keytab
--principal=oryx/127.5.22.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.61:39427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.22.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
W20260928 06:35:42.317874 11843 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.5.22.62:38463,127.5.22.61:39427'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 06:35:42.318804 11843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:42.319868 11843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:35:42.328276 11843 server_base.cc:1034] running on GCE node
W20260928 06:35:42.328362 11851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:42.328280 11849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:42.328532 11848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:42.329041 11843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:42.330121 11843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:42.331296 11843 hybrid_clock.cc:648] HybridClock initialized: now 1790577342331277 us; error 31 us; skew 500 ppm
Sep 28 06:35:42 dist-test-slave-j0qf krb5kdc[11334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577342, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:42.333935 11843 init.cc:377] Logged in from keytab as oryx/127.5.22.61@KRBTEST.COM (short username oryx)
I20260928 06:35:42.334825 11843 webserver.cc:460] Webserver started at http://127.5.22.61:36507/ using document root <none> and password file <none>
I20260928 06:35:42.335366 11843 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:42.335451 11843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:42.335721 11843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:42.337680 11843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/data/instance:
uuid: "2181a9b0ee3a437693d113b48de29cef"
format_stamp: "Formatted at 2026-09-28 06:35:42 on dist-test-slave-j0qf"
server_key: "24eab6b30b954c2e81693d5c6c92463e"
server_key_iv: "5ee2c246a9333ba33a3d9d35691ef4a1"
server_key_version: "encryptionkey@0"
I20260928 06:35:42.338233 11843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/wal/instance:
uuid: "2181a9b0ee3a437693d113b48de29cef"
format_stamp: "Formatted at 2026-09-28 06:35:42 on dist-test-slave-j0qf"
server_key: "24eab6b30b954c2e81693d5c6c92463e"
server_key_iv: "5ee2c246a9333ba33a3d9d35691ef4a1"
server_key_version: "encryptionkey@0"
I20260928 06:35:42.341789 11843 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:42.344110 11858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:42.345301 11843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:42.345429 11843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/wal
uuid: "2181a9b0ee3a437693d113b48de29cef"
format_stamp: "Formatted at 2026-09-28 06:35:42 on dist-test-slave-j0qf"
server_key: "24eab6b30b954c2e81693d5c6c92463e"
server_key_iv: "5ee2c246a9333ba33a3d9d35691ef4a1"
server_key_version: "encryptionkey@0"
I20260928 06:35:42.345531 11843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:42.368036 11843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:42.371140 11843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:42.371348 11843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:42.378383 11843 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.61:39427
I20260928 06:35:42.378656 11910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.61:39427 every 8 connection(s)
I20260928 06:35:42.379374 11843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/data/info.pb
I20260928 06:35:42.382138 11911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:42.384193 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 11843
I20260928 06:35:42.386037 11911 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:35:42 dist-test-slave-j0qf krb5kdc[11334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577342, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.61@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM
Sep 28 06:35:42 dist-test-slave-j0qf krb5kdc[11334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.22.61@KRBTEST.COM
I20260928 06:35:42.398152 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.387739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48188 (local address 127.5.22.62:38463)
0928 06:35:42.388152 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.388156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.388448 (+ 292us) server_negotiation.cc:407] Connection header received
0928 06:35:42.388613 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.388616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.388666 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.388755 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:42.389398 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.390084 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.391435 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.391763 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.391877 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.394233 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:42.394241 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:42.394243 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:42.394267 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:42.395874 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.396471 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.396475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.396477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.396575 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.396963 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.396967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.396968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.397273 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:42.397351 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.397834 (+ 483us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.397967 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319}
I20260928 06:35:42.399205 11915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.387654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.0.0.1:48188)
0928 06:35:42.388310 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:42.388360 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:42.388491 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:42.388806 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:42.388810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:42.389120 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:42.389263 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.389271 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.390199 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.390203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.391310 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.391320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.391912 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.391916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.392014 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.392561 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:42.392576 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:42.394084 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:42.395984 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.395993 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.396006 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.396378 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.396676 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.396678 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.396679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.396845 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.397367 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:42.397372 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:42.397606 (+ 234us) client_negotiation.cc:769] Sending connection context
0928 06:35:42.397766 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 06:35:42.398154 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":89,"spinlock_wait_cycles":1792,"thread_start_us":121,"threads_started":1}
I20260928 06:35:42.399206 11914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.385765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36428 (local address 127.5.22.61:39427)
0928 06:35:42.386177 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.386192 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.386228 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:35:42.386939 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.386956 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.387297 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.387647 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:42.388589 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.389806 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.390920 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.391437 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.391594 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.394701 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:42.394707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:42.394716 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:42.394736 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:42.396329 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.396805 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.396810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.396812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.396885 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.397169 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.397171 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.397172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.397600 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:42.397804 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.398029 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.398200 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":279,"thread_start_us":133,"threads_started":1}
I20260928 06:35:42.400512 11911 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:35:42 dist-test-slave-j0qf krb5kdc[11334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577342, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.61@KRBTEST.COM for oryx/127.5.22.61@KRBTEST.COM
I20260928 06:35:42.408181 11915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.400983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:39427 (local address 127.0.0.1:36434)
0928 06:35:42.401157 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:42.401173 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:42.401255 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:42.401516 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:42.401520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:42.401675 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:42.401760 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.401765 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.402464 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.402466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.402960 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.402967 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.403788 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.403790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.403859 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.404321 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:42.404332 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:42.405454 (+ 1122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:42.406944 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.406946 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.406947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.407141 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.407357 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.407359 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.407360 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.407431 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.407780 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:42.407782 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:42.407851 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 06:35:42.407931 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 06:35:42.408034 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 06:35:42.408200 11914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.401056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36434 (local address 127.5.22.61:39427)
0928 06:35:42.401168 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.401174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.401229 (+ 55us) server_negotiation.cc:407] Connection header received
0928 06:35:42.401335 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.401338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.401395 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.401476 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:42.401851 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.402374 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.403052 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.403698 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.403776 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.405597 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:42.405600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:42.405602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:42.405622 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:42.406843 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.407219 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.407222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.407223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.407286 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.407498 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.407500 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.407501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.407674 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:42.407747 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.407978 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.408084 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":36}
I20260928 06:35:42.411032 11911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef: Bootstrap starting.
I20260928 06:35:42.413301 11911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:42.414033 11911 log.cc:824] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:42.416030 11911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef: No bootstrap required, opened a new log
I20260928 06:35:42.419078 11911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } }
I20260928 06:35:42.419266 11911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:42.419317 11911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2181a9b0ee3a437693d113b48de29cef, State: Initialized, Role: FOLLOWER
I20260928 06:35:42.419751 11911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } }
I20260928 06:35:42.420135 11917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } } }
I20260928 06:35:42.420253 11917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:35:42.420805 11911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [sys.catalog]: configured and running, proceeding with master startup.
W20260928 06:35:42.425402 11928 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:35:42.425489 11928 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:35:42.427434 11843 master_runner.cc:428] Detected that this master 2181a9b0ee3a437693d113b48de29cef is joining an existing cluster
I20260928 06:35:42.427490 11843 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.22.61:39427
I20260928 06:35:42.437590 11914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.429733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36444 (local address 127.5.22.61:39427)
0928 06:35:42.429853 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.429858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.429873 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:35:42.429984 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.429987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.430027 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.430111 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:42.430702 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.431521 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.432752 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.433040 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.433142 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.434114 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:42.434127 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:42.434130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:42.434153 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:42.435855 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.436338 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.436343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.436345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.436438 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.436716 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.436718 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.436720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.436927 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:42.437030 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.437290 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.437434 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":46,"spinlock_wait_cycles":1920}
I20260928 06:35:42.437747 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.429379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48196 (local address 127.5.22.62:38463)
0928 06:35:42.429509 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.429514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.429791 (+ 277us) server_negotiation.cc:407] Connection header received
0928 06:35:42.429964 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.429968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.430022 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.430115 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:42.430790 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.431586 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.432478 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.432731 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.432839 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.434178 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:42.434184 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:42.434190 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:42.434217 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:42.435952 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.436468 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.436472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.436474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.436558 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.436853 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.436857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.436860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.437088 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:42.437191 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.437460 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.437568 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 06:35:42.437597 11934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.429418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:39427 (local address 127.0.0.1:36444)
0928 06:35:42.429805 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:42.429817 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:42.429882 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:42.430135 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:42.430138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:42.430455 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:42.430605 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.430615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.431649 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.431653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.432639 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.432651 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.433142 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.433145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.433233 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.433623 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:42.433634 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:42.433987 (+ 353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:42.435974 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.435980 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.435985 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.436249 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.436523 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.436526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.436528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.436621 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.437011 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:42.437014 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:42.437092 (+ 78us) client_negotiation.cc:769] Sending connection context
0928 06:35:42.437226 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 06:35:42.437373 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":31,"thread_start_us":105,"threads_started":1}
I20260928 06:35:42.438261 11935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.429331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.0.0.1:48196)
0928 06:35:42.429708 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:42.429724 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:42.429866 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:42.430239 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:42.430242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:42.430543 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:42.430681 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.430689 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.431675 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.431678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.432347 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.432365 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.432849 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.432852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.432938 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.433476 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:42.433497 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:42.434064 (+ 567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:42.436064 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.436068 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.436070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.436369 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.436652 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.436656 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.436658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.436752 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.437191 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:42.437197 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:42.437275 (+ 78us) client_negotiation.cc:769] Sending connection context
0928 06:35:42.437923 (+ 648us) client_negotiation.cc:240] Negotiation successful
0928 06:35:42.438065 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":22,"spinlock_wait_cycles":768,"thread_start_us":100,"threads_started":1}
Sep 28 06:35:42 dist-test-slave-j0qf krb5kdc[11334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577341, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.62@KRBTEST.COM for oryx/127.5.22.61@KRBTEST.COM
I20260928 06:35:42.451290 11914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.442393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36448 (local address 127.5.22.61:39427)
0928 06:35:42.442505 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.442509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.442847 (+ 338us) server_negotiation.cc:407] Connection header received
0928 06:35:42.443059 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.443062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.443116 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.443201 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:42.443802 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.444558 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.445496 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.445712 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.445813 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.447915 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:42.447918 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:42.447920 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:42.447939 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:42.449516 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.450005 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.450010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.450012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.450093 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.450415 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.450419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.450421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.450594 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:42.450696 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.451021 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.451133 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:35:42.451321 11936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.442358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:39427 (local address 127.0.0.1:36448)
0928 06:35:42.442717 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:42.442773 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:42.442899 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:42.443237 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:42.443242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:42.443605 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:42.443701 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.443707 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.444679 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.444682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.445375 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.445384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.445815 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.445819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.445927 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.446419 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:42.446430 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:42.447781 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:42.449655 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.449663 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.449669 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.449888 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.450188 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.450192 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.450194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.450278 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.450696 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:42.450704 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:42.450822 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 06:35:42.450957 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 06:35:42.451157 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":98,"threads_started":1}
I20260928 06:35:42.453018 11376 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.22.61:39427: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2181a9b0ee3a437693d113b48de29cef"
member_type: NON_VOTER
last_known_addr {
host: "127.5.22.61"
port: 39427
}
attrs {
promote: true
}
}
dest_uuid: "5973a583576742e5a79e1b84b9e53783"
cas_config_opid_index: -1
I20260928 06:35:42.453734 11376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: NON_VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: true } }
I20260928 06:35:42.455929 11939 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2181a9b0ee3a437693d113b48de29cef (127.5.22.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: NON_VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: true } } }
I20260928 06:35:42.456508 11885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:42.457441 11939 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.22.61:39427
I20260928 06:35:42.457659 11939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5973a583576742e5a79e1b84b9e53783" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: NON_VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: true } } }
I20260928 06:35:42.457753 11939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:42.457845 11885 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [term 1 FOLLOWER]: Refusing update from remote peer 5973a583576742e5a79e1b84b9e53783: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 06:35:42.457891 11843 master.cc:537] Master@127.5.22.61:39427 shutting down...
I20260928 06:35:42.458467 11939 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: NON_VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 06:35:42.460175 11363 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 06:35:42.461205 11363 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 -> Peer 2181a9b0ee3a437693d113b48de29cef (127.5.22.61:39427): Couldn't send request to peer 2181a9b0ee3a437693d113b48de29cef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 06:35:42.462464 11843 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:42.462749 11843 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:42.462864 11843 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef: stopping tablet replica
I20260928 06:35:42.467344 11843 master.cc:552] Master@127.5.22.61:39427 shutdown complete.
I20260928 06:35:42.467411 11843 master_runner.cc:305] Clearing existing system tablet
I20260928 06:35:42.467633 11843 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:42.467702 11843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
WARNING: no policy specified for oryx/127.5.22.60@KRBTEST.COM; defaulting to no policy
I20260928 06:35:42.468911 11843 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:35:42.469851 11943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:42.470046 11843 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:42.470183 11843 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:42.470222 11843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/wal
uuid: "2181a9b0ee3a437693d113b48de29cef"
format_stamp: "Formatted at 2026-09-28 06:35:42 on dist-test-slave-j0qf"
server_key: "24eab6b30b954c2e81693d5c6c92463e"
server_key_iv: "5ee2c246a9333ba33a3d9d35691ef4a1"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.22.60@KRBTEST.COM" created.
I20260928 06:35:42.472718 11843 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:42.475277 11843 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 06:35:42.475394 11843 log.cc:1196] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 06:35:42.475654 11843 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef: Deleting consensus metadata
I20260928 06:35:42.475843 11843 master_runner.cc:315] Copying system tablet from 127.5.22.62:38463
I20260928 06:35:42.477070 11843 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef: tablet copy: Beginning tablet copy session from remote peer at address 127.5.22.62:38463
I20260928 06:35:42.485085 11950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.477655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.0.0.1:48202)
0928 06:35:42.477948 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:42.477960 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:42.478084 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:42.478362 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:42.478365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:42.478580 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:42.478660 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.478666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.479421 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.479426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.480148 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.480157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.480610 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.480613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.480694 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.481232 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:42.481247 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:42.481578 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:42.483427 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.483432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.483434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.483707 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.483986 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.483989 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.483991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.484071 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.484529 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:42.484532 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:42.484629 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 06:35:42.484731 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 06:35:42.484875 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":80,"threads_started":1}
I20260928 06:35:42.485101 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.477707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48202 (local address 127.5.22.62:38463)
0928 06:35:42.477834 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.477840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.478039 (+ 199us) server_negotiation.cc:407] Connection header received
0928 06:35:42.478170 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.478173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.478218 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.478309 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:42.478776 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.479316 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.480267 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.480495 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.480607 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.481711 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:42.481714 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:42.481717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:42.481741 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:42.483328 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.483805 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.483810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.483812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.483894 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.484155 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.484160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.484161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.484385 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:42.484532 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.484796 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.484943 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:35:42.486022 11399 tablet_copy_service.cc:140] P 5973a583576742e5a79e1b84b9e53783: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2181a9b0ee3a437693d113b48de29cef ({username='oryx', principal='oryx/127.5.22.61@KRBTEST.COM'} at 127.0.0.1:48202)
I20260928 06:35:42.486184 11399 tablet_copy_service.cc:161] P 5973a583576742e5a79e1b84b9e53783: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2181a9b0ee3a437693d113b48de29cef at {username='oryx', principal='oryx/127.5.22.61@KRBTEST.COM'} at 127.0.0.1:48202: session id = 2181a9b0ee3a437693d113b48de29cef-00000000000000000000000000000000
I20260928 06:35:42.487869 11399 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 06:35:42.489521 11843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.22.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.60.keytab.
Entry for principal oryx/127.5.22.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.60.keytab.
I20260928 06:35:42.492329 11843 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef: tablet copy: Starting download of 0 data blocks...
I20260928 06:35:42.492527 11843 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef: tablet copy: Starting download of 1 WAL segments...
I20260928 06:35:42.493348 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.60:46045
--webserver_interface=127.5.22.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.60.keytab
--principal=oryx/127.5.22.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:38463,127.5.22.61:39427,127.5.22.60:46045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:35:42.494445 11843 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 06:35:42.496356 11399 tablet_copy_service.cc:342] P 5973a583576742e5a79e1b84b9e53783: Request end of tablet copy session 2181a9b0ee3a437693d113b48de29cef-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.22.61@KRBTEST.COM'} at 127.0.0.1:48202
I20260928 06:35:42.496490 11399 tablet_copy_service.cc:433] P 5973a583576742e5a79e1b84b9e53783: ending tablet copy session 2181a9b0ee3a437693d113b48de29cef-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2181a9b0ee3a437693d113b48de29cef
I20260928 06:35:42.504962 11843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:42.506657 11952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:42.506696 11955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:42.507254 11953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:42.507325 11843 server_base.cc:1034] running on GCE node
I20260928 06:35:42.507525 11843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:42.507814 11843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:42.508950 11843 hybrid_clock.cc:648] HybridClock initialized: now 1790577342508927 us; error 40 us; skew 500 ppm
Sep 28 06:35:42 dist-test-slave-j0qf krb5kdc[11334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577342, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:42.510620 11843 init.cc:377] Logged in from keytab as oryx/127.5.22.61@KRBTEST.COM (short username oryx)
I20260928 06:35:42.511101 11843 webserver.cc:460] Webserver started at http://127.5.22.61:37693/ using document root <none> and password file <none>
I20260928 06:35:42.511297 11843 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:42.511343 11843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:42.512702 11843 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:42.513742 11962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:42.513991 11843 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:42.514065 11843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/wal
uuid: "2181a9b0ee3a437693d113b48de29cef"
format_stamp: "Formatted at 2026-09-28 06:35:42 on dist-test-slave-j0qf"
server_key: "24eab6b30b954c2e81693d5c6c92463e"
server_key_iv: "5ee2c246a9333ba33a3d9d35691ef4a1"
server_key_version: "encryptionkey@0"
I20260928 06:35:42.514142 11843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:42.555935 11843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:42.558180 11843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:42.563514 11843 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.61:39427
I20260928 06:35:42.563712 12014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.61:39427 every 8 connection(s)
I20260928 06:35:42.564143 11843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-1/data/info.pb
I20260928 06:35:42.565857 12015 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:35:42.566883 12015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef: Bootstrap starting.
I20260928 06:35:42.577098 12015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:35:42.577423 12015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef: Bootstrap complete.
I20260928 06:35:42.578047 12015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: NON_VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: true } }
I20260928 06:35:42.578205 12015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2181a9b0ee3a437693d113b48de29cef, State: Initialized, Role: LEARNER
I20260928 06:35:42.578348 12015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: NON_VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: true } }
I20260928 06:35:42.578833 12015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:42.578735 12018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: NON_VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: true } } }
I20260928 06:35:42.579150 12018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [sys.catalog]: This master's current role is: LEARNER
I20260928 06:35:42.582154 12029 catalog_manager.cc:1259] Loaded cluster ID: 6c9264db8f9d43f884348bed61629dcc
I20260928 06:35:42.582268 12029 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef: loading cluster ID for follower catalog manager: success
I20260928 06:35:42.584543 12029 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef: acquiring CA information for follower catalog manager: success
I20260928 06:35:42.586017 12029 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 06:35:42.617645 11951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:42.617919 11951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:42.618001 11951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:42.622205 11951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:42.622285 11951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:42.622316 11951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:42.622375 11951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:42.622397 11951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:42.627945 11951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:38463,127.5.22.61:39427,127.5.22.60:46045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.60.keytab
--principal=oryx/127.5.22.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.60:46045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.22.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:42.629029 11951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:42.630152 11951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:42.639278 12034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:42.639274 12035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:42.639252 12037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:42.639353 11951 server_base.cc:1034] running on GCE node
I20260928 06:35:42.640086 11951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:42.641198 11951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:42.642452 11951 hybrid_clock.cc:648] HybridClock initialized: now 1790577342642379 us; error 83 us; skew 500 ppm
Sep 28 06:35:42 dist-test-slave-j0qf krb5kdc[11334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577342, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:42.645244 11951 init.cc:377] Logged in from keytab as oryx/127.5.22.60@KRBTEST.COM (short username oryx)
I20260928 06:35:42.646136 11951 webserver.cc:460] Webserver started at http://127.5.22.60:42129/ using document root <none> and password file <none>
I20260928 06:35:42.646730 11951 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:42.646795 11951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:42.646962 11951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:42.649138 11951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/data/instance:
uuid: "032daa7770c44a8aa5d0171610287234"
format_stamp: "Formatted at 2026-09-28 06:35:42 on dist-test-slave-j0qf"
server_key: "b03f607f0d25627615779c7ef5fad37a"
server_key_iv: "aed77a5d33bb86a30c59a008a3ceae95"
server_key_version: "encryptionkey@0"
I20260928 06:35:42.649690 11951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/wal/instance:
uuid: "032daa7770c44a8aa5d0171610287234"
format_stamp: "Formatted at 2026-09-28 06:35:42 on dist-test-slave-j0qf"
server_key: "b03f607f0d25627615779c7ef5fad37a"
server_key_iv: "aed77a5d33bb86a30c59a008a3ceae95"
server_key_version: "encryptionkey@0"
I20260928 06:35:42.653959 11951 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260928 06:35:42.656342 12044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:42.657717 11951 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 06:35:42.657820 11951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/wal
uuid: "032daa7770c44a8aa5d0171610287234"
format_stamp: "Formatted at 2026-09-28 06:35:42 on dist-test-slave-j0qf"
server_key: "b03f607f0d25627615779c7ef5fad37a"
server_key_iv: "aed77a5d33bb86a30c59a008a3ceae95"
server_key_version: "encryptionkey@0"
I20260928 06:35:42.657896 11951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:42.698065 11951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:42.701507 11951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:42.701691 11951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:42.709030 11951 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.60:46045
I20260928 06:35:42.709349 12096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.60:46045 every 8 connection(s)
I20260928 06:35:42.710021 11951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/data/info.pb
I20260928 06:35:42.711122 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 11951
I20260928 06:35:42.714003 12097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:42.718541 12097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:35:42 dist-test-slave-j0qf krb5kdc[11334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.22.60@KRBTEST.COM
Sep 28 06:35:42 dist-test-slave-j0qf krb5kdc[11334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577342, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.60@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM
I20260928 06:35:42.726445 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.712704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39318 (local address 127.5.22.60:46045)
0928 06:35:42.713073 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.713088 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.713123 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:35:42.713776 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.713791 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.714157 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.714561 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:42.715504 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.716647 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.717800 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.718259 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.718440 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.721616 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:42.721621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:42.721632 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:42.721659 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:42.723485 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.723941 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.723946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.723954 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.724067 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.724358 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.724362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.724363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.724811 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:42.725003 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.725303 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.725572 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":107,"threads_started":1}
I20260928 06:35:42.726444 11570 heartbeater.cc:503] Master 127.5.22.62:38463 was elected leader, sending a full tablet report...
I20260928 06:35:42.730206 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.720667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.0.0.1:48214)
0928 06:35:42.721161 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:42.721199 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:42.721344 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:42.721626 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:42.721631 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:42.722005 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:42.722124 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.722130 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.722937 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.722944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.723597 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.723606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.724071 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.724075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.724194 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.724693 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:42.724704 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:42.726053 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:42.728704 (+ 2651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.728710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.728715 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.728925 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.729224 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.729226 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.729228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.729311 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.729700 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:42.729704 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:42.729799 (+ 95us) client_negotiation.cc:769] Sending connection context
0928 06:35:42.729923 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 06:35:42.730057 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":130,"threads_started":1}
I20260928 06:35:42.730213 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.720779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48214 (local address 127.5.22.62:38463)
0928 06:35:42.720909 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.720912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.721282 (+ 370us) server_negotiation.cc:407] Connection header received
0928 06:35:42.721444 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.721447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.721502 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.721595 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:42.722254 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.722828 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.723723 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.723955 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.724072 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.726188 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:42.726192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:42.726195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:42.726247 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:42.728589 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.729023 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.729025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.729027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.729106 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.729405 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.729407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.729409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.729588 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:42.729676 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.730003 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.730101 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 06:35:42.731443 12097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:35:42 dist-test-slave-j0qf krb5kdc[11334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577342, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.60@KRBTEST.COM for oryx/127.5.22.61@KRBTEST.COM
I20260928 06:35:42.739442 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.731861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:39427 (local address 127.0.0.1:36458)
0928 06:35:42.731980 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:42.731992 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:42.732061 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:42.732429 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:42.732431 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:42.732558 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:42.732644 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.732649 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.733594 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.733596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.734112 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.734119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.734657 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.734659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.734729 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.735125 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:42.735135 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:42.736278 (+ 1143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:42.738066 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.738068 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.738070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.738244 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.738644 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.738646 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.738648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.738721 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.739076 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:42.739077 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:42.739150 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 06:35:42.739230 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 06:35:42.739327 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 06:35:42.739727 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.731989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36458 (local address 127.5.22.61:39427)
0928 06:35:42.732226 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.732229 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.732242 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:35:42.732277 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.732279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.732315 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.732403 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:42.732787 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.733495 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.734228 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.734560 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.734646 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.736411 (+ 1765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:42.736414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:42.736416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:42.736484 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:42.737951 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.738413 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.738416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.738417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.738553 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.738806 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.738808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.738810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.738987 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:42.739066 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.739291 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.739389 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":102,"threads_started":1}
I20260928 06:35:42.740708 12097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46045 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:35:42 dist-test-slave-j0qf krb5kdc[11334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577342, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.60@KRBTEST.COM for oryx/127.5.22.60@KRBTEST.COM
I20260928 06:35:42.748494 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.741165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:46045 (local address 127.0.0.1:39326)
0928 06:35:42.741277 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:42.741293 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:42.741369 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:42.741618 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:42.741620 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:42.741744 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:42.741814 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.741818 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.742998 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.743001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.743494 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.743502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.743924 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.743926 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.743982 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.744417 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:42.744428 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:42.745587 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:42.747261 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.747263 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.747264 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.747438 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.747736 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.747738 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.747739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.747808 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.748138 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:42.748140 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:42.748210 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 06:35:42.748289 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:35:42.748381 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 06:35:42.748658 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.741254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39326 (local address 127.5.22.60:46045)
0928 06:35:42.741444 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.741447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.741460 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:35:42.741488 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.741490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.741523 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.741590 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:42.741945 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.742898 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.743608 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.743825 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.743907 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.745708 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:42.745711 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:42.745713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:42.745733 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:42.747170 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.747567 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.747570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.747571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.747647 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.747889 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.747892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.747893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.748057 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:42.748134 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.748381 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.748484 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:35:42.751602 12097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234: Bootstrap starting.
I20260928 06:35:42.753798 12097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:42.754551 12097 log.cc:824] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:42.756431 12097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234: No bootstrap required, opened a new log
I20260928 06:35:42.759486 12097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46045 } }
I20260928 06:35:42.759697 12097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:42.759732 12097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032daa7770c44a8aa5d0171610287234, State: Initialized, Role: FOLLOWER
I20260928 06:35:42.760107 12097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46045 } }
I20260928 06:35:42.760519 12103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46045 } } }
I20260928 06:35:42.760792 12103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:35:42.761303 12097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 06:35:42.766566 12114 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:35:42.766655 12114 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:35:42.770527 11951 master_runner.cc:428] Detected that this master 032daa7770c44a8aa5d0171610287234 is joining an existing cluster
I20260928 06:35:42.770604 11951 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.22.60:46045
I20260928 06:35:42.779847 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 11441
I20260928 06:35:42.782469 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.773527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39340 (local address 127.5.22.60:46045)
0928 06:35:42.773641 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.773645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.774112 (+ 467us) server_negotiation.cc:407] Connection header received
0928 06:35:42.774243 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.774246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.774280 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.774347 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:42.774910 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.775659 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.776847 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.777179 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.777248 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.778835 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:42.778839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:42.778842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:42.778903 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:42.780604 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.781301 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.781305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.781307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.781375 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.781621 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.781624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.781626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.781809 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:42.781877 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.782172 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.782282 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:35:42.782881 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.773439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:46045 (local address 127.0.0.1:39340)
0928 06:35:42.774044 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:42.774056 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:42.774157 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:42.774473 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:42.774475 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:42.774683 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:42.774794 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.774801 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.775808 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.775812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.776723 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.776733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.777395 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.777399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.777664 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.778157 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:42.778172 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:42.778680 (+ 508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:42.780915 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.780918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.780921 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.781202 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.781463 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.781466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.781468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.781540 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.781931 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:42.781934 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:42.782025 (+ 91us) client_negotiation.cc:769] Sending connection context
0928 06:35:42.782598 (+ 573us) client_negotiation.cc:240] Negotiation successful
0928 06:35:42.782721 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":137,"thread_start_us":84,"threads_started":1}
I20260928 06:35:42.787396 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:44565
--local_ip_for_outbound_sockets=127.5.22.1
--tserver_master_addrs=127.5.22.62:38463
--webserver_port=39681
--webserver_interface=127.5.22.1
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.1.keytab
--principal=oryx/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.22.62:38463,127.5.22.61:39427
--tserver_master_addrs=127.5.22.62:38463,127.5.22.61:39427,127.5.22.60:46045 with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:35:42.808175 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.773077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48228 (local address 127.5.22.62:38463)
0928 06:35:42.773210 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.773216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.773850 (+ 634us) server_negotiation.cc:407] Connection header received
0928 06:35:42.773951 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.773955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.774009 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.774082 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:42.774677 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.775427 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.776808 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.777147 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.777222 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.803870 (+ 26648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:42.803875 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:42.803878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:42.803912 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:42.806187 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.806773 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.806777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.806779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.806865 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.807169 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.807172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.807174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.807406 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:42.807488 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.807846 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.807998 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 06:35:42.808192 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.772600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:39427 (local address 127.0.0.1:36460)
0928 06:35:42.773346 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:42.773358 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:42.773531 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:42.773916 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:42.773921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:42.774164 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:42.774349 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.774360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.775464 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.775469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.776670 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.776680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.777395 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.777399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.777593 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.803805 (+ 26212us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:42.803822 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:42.804313 (+ 491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:42.806481 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.806484 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.806486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.806778 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.807104 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.807107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.807109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.807190 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.807618 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:42.807621 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:42.807707 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 06:35:42.807807 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 06:35:42.807984 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":215,"thread_start_us":294,"threads_started":1}
I20260928 06:35:42.809002 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.773075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.0.0.1:48228)
0928 06:35:42.773746 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:42.773760 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:42.773861 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:42.774203 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:42.774205 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:42.774425 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:42.774555 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.774561 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.775568 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.775570 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.776670 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.776680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.777395 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.777399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.777633 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.803106 (+ 25473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:42.803129 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:42.803626 (+ 497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:42.806357 (+ 2731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.806361 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.806365 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.806650 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.806972 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.806975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.806977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.807068 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.807548 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:42.807554 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:42.807657 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 06:35:42.808646 (+ 989us) client_negotiation.cc:240] Negotiation successful
0928 06:35:42.808812 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":202,"thread_start_us":107,"threads_started":1}
I20260928 06:35:42.809016 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.772747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36460 (local address 127.5.22.61:39427)
0928 06:35:42.772909 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.772915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.773445 (+ 530us) server_negotiation.cc:407] Connection header received
0928 06:35:42.773650 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.773655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.773714 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.773756 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:42.774485 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.775327 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.776807 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.777154 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.777222 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.804522 (+ 27300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:42.804527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:42.804530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:42.804564 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:42.806333 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.806910 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.806916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.806918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.807002 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.807282 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.807285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.807287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.807492 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:42.807577 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.808730 (+ 1153us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.808861 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 06:35:42 dist-test-slave-j0qf krb5kdc[11334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577341, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.62@KRBTEST.COM for oryx/127.5.22.60@KRBTEST.COM
I20260928 06:35:42.823199 11936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.813239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:46045 (local address 127.0.0.1:39350)
0928 06:35:42.813442 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:42.813457 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:42.813536 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:42.813945 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:42.813948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:42.814172 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:42.814282 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.814290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.815344 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.815347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.816299 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.816310 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.816861 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.816864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.816968 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.817464 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:42.817478 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:42.819259 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:42.821229 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.821232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.821234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.821489 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.821799 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.821803 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.821810 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.822121 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.822602 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:42.822605 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:42.822718 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 06:35:42.822844 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 06:35:42.823015 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260928 06:35:42.823203 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.813313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39350 (local address 127.5.22.60:46045)
0928 06:35:42.813691 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.813695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.813707 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:35:42.813770 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.813773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.813823 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.813906 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:42.814446 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.815228 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.816433 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.816745 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.816849 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.819452 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:42.819457 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:42.819460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:42.819487 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:42.821108 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.821595 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.821599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.821601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.821685 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.822241 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.822244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.822246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.822491 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:42.822589 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.822931 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.823047 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295}
I20260928 06:35:42.824666 11376 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.22.60:46045: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "032daa7770c44a8aa5d0171610287234"
member_type: NON_VOTER
last_known_addr {
host: "127.5.22.60"
port: 46045
}
attrs {
promote: true
}
}
dest_uuid: "5973a583576742e5a79e1b84b9e53783"
cas_config_opid_index: 5
I20260928 06:35:42.825260 11376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: NON_VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: true } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: NON_VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: true } }
I20260928 06:35:42.826660 11939 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 032daa7770c44a8aa5d0171610287234 (127.5.22.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: NON_VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: true } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: NON_VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: true } } }
I20260928 06:35:42.828055 11939 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.22.60:46045
I20260928 06:35:42.828222 11939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5973a583576742e5a79e1b84b9e53783" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: NON_VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: true } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: NON_VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: true } } }
I20260928 06:35:42.828316 11939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:42.829226 12071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:42.830631 12071 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [term 1 FOLLOWER]: Refusing update from remote peer 5973a583576742e5a79e1b84b9e53783: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 06:35:42.831312 11938 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: NON_VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:42.832422 11951 master.cc:537] Master@127.5.22.60:46045 shutting down...
I20260928 06:35:42.833997 11936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.826439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:39427 (local address 127.0.0.1:36476)
0928 06:35:42.826573 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:42.826589 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:42.826674 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:42.829054 (+ 2380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:42.829059 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:42.829080 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:42.829172 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.829180 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.830554 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.830558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.832209 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.832221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.833543 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.833548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.833650 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.833703 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 06:35:42.833782 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:35:42.833856 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 06:35:42.834569 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.828521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36476 (local address 127.5.22.61:39427)
0928 06:35:42.828678 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.828683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.828701 (+ 18us) server_negotiation.cc:407] Connection header received
0928 06:35:42.828753 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.828756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.828816 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.828915 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:42.829399 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.830271 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.832425 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.833142 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.833266 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.833414 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.834270 (+ 856us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.834394 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 06:35:42.837568 12071 raft_consensus.cc:858] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46045 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: NON_VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: NON_VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: true } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: NON_VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: true } }
W20260928 06:35:42.837664 12071 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46045 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: NON_VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: NON_VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: true } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: NON_VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: true } }
I20260928 06:35:42.839033 12071 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2181a9b0ee3a437693d113b48de29cef (127.5.22.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: NON_VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: true } } }
I20260928 06:35:42.840564 12124 mvcc.cc:205] Tried to move back new op lower bound from 7334204790742589440 to 7334204790649495552. Current Snapshot: MvccSnapshot[applied={T|T < 7334204790682648576}]
I20260928 06:35:42.841115 11989 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [term 1 LEARNER]: Refusing update from remote peer 5973a583576742e5a79e1b84b9e53783: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 06:35:42.841212 12103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5973a583576742e5a79e1b84b9e53783. Latest consensus state: current_term: 1 leader_uuid: "5973a583576742e5a79e1b84b9e53783" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: NON_VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: true } } }
I20260928 06:35:42.841420 12103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 06:35:42.841576 11938 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: NON_VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 06:35:42.841701 12103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5973a583576742e5a79e1b84b9e53783" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: NON_VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: true } } }
I20260928 06:35:42.841777 12103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 06:35:42.841842 12103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5973a583576742e5a79e1b84b9e53783" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: NON_VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: true } } }
I20260928 06:35:42.841894 12103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 06:35:42.843415 11989 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 032daa7770c44a8aa5d0171610287234 (127.5.22.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: NON_VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: true } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: NON_VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: true } } }
I20260928 06:35:42.845278 12132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5973a583576742e5a79e1b84b9e53783" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: NON_VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: true } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: NON_VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: true } } }
I20260928 06:35:42.845530 12132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [sys.catalog]: This master's current role is: LEARNER
I20260928 06:35:42.845737 12018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5973a583576742e5a79e1b84b9e53783. Latest consensus state: current_term: 1 leader_uuid: "5973a583576742e5a79e1b84b9e53783" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: NON_VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: true } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: NON_VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: true } } }
I20260928 06:35:42.845870 12018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [sys.catalog]: This master's current role is: LEARNER
I20260928 06:35:42.848368 11938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5973a583576742e5a79e1b84b9e53783" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: NON_VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: true } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: NON_VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: true } } }
I20260928 06:35:42.848503 11938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:42.849645 11951 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
W20260928 06:35:42.849658 11366 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 -> Peer 032daa7770c44a8aa5d0171610287234 (127.5.22.60:46045): Couldn't send request to peer 032daa7770c44a8aa5d0171610287234. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 06:35:42.849915 11951 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260928 06:35:42.850006 11951 tablet_replica.cc:330] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234: stopping tablet replica
I20260928 06:35:42.850929 11939 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783: attempting to promote NON_VOTER 2181a9b0ee3a437693d113b48de29cef to VOTER
I20260928 06:35:42.851378 11939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: NON_VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: true } }
I20260928 06:35:42.853322 11989 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [term 1 LEARNER]: Refusing update from remote peer 5973a583576742e5a79e1b84b9e53783: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 06:35:42.853694 12133 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260928 06:35:42.854252 11366 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 -> Peer 032daa7770c44a8aa5d0171610287234 (127.5.22.60:46045): Couldn't send request to peer 032daa7770c44a8aa5d0171610287234. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 06:35:42.856431 11938 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2181a9b0ee3a437693d113b48de29cef (127.5.22.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: NON_VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: true } } }
I20260928 06:35:42.857914 11938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5973a583576742e5a79e1b84b9e53783" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: NON_VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: true } } }
I20260928 06:35:42.858021 11938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:42.858134 11938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5973a583576742e5a79e1b84b9e53783" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: NON_VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: true } } }
I20260928 06:35:42.858207 11938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:42.858796 11989 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2181a9b0ee3a437693d113b48de29cef (127.5.22.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: NON_VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: true } } }
I20260928 06:35:42.860226 12018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5973a583576742e5a79e1b84b9e53783" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: NON_VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: true } } }
I20260928 06:35:42.860328 12018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:35:42.867522 11951 master.cc:552] Master@127.5.22.60:46045 shutdown complete.
I20260928 06:35:42.867586 11951 master_runner.cc:305] Clearing existing system tablet
I20260928 06:35:42.867910 11951 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:42.867973 11951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:42.869354 11951 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:42.870476 12135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:42.870749 11951 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:42.870885 11951 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:42.870922 11951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/wal
uuid: "032daa7770c44a8aa5d0171610287234"
format_stamp: "Formatted at 2026-09-28 06:35:42 on dist-test-slave-j0qf"
server_key: "b03f607f0d25627615779c7ef5fad37a"
server_key_iv: "aed77a5d33bb86a30c59a008a3ceae95"
server_key_version: "encryptionkey@0"
I20260928 06:35:42.872900 11951 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:42.875684 11951 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 06:35:42.875783 11951 log.cc:1196] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 06:35:42.876088 11951 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234: Deleting consensus metadata
I20260928 06:35:42.876267 11951 master_runner.cc:315] Copying system tablet from 127.5.22.62:38463
I20260928 06:35:42.877521 11951 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234: tablet copy: Beginning tablet copy session from remote peer at address 127.5.22.62:38463
I20260928 06:35:42.887585 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.878098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.0.0.1:48236)
0928 06:35:42.878461 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:42.878475 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:42.878617 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:42.878962 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:42.878965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:42.879205 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:42.879329 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.879340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.880591 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.880595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.881492 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.881503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.882228 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.882233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.882371 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.882975 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:42.882989 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:42.883436 (+ 447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:42.885821 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.885824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.885826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.886089 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.886479 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:42.886483 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:42.886485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:42.886580 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:42.887039 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:42.887042 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:42.887146 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 06:35:42.887261 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 06:35:42.887408 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":100,"threads_started":1}
I20260928 06:35:42.887712 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.878221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48236 (local address 127.5.22.62:38463)
0928 06:35:42.878377 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.878383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.878567 (+ 184us) server_negotiation.cc:407] Connection header received
0928 06:35:42.878740 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.878743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.878804 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.878898 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:42.879505 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.880452 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.881650 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.882053 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.882167 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.883671 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:42.883675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:42.883679 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:42.883707 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:42.885696 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.886202 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.886207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.886209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.886307 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:42.886686 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:42.886691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:42.886693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:42.886925 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:42.887032 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.887336 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.887473 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:35:42.888525 11399 tablet_copy_service.cc:140] P 5973a583576742e5a79e1b84b9e53783: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 032daa7770c44a8aa5d0171610287234 ({username='oryx', principal='oryx/127.5.22.60@KRBTEST.COM'} at 127.0.0.1:48236)
I20260928 06:35:42.888633 11399 tablet_copy_service.cc:161] P 5973a583576742e5a79e1b84b9e53783: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 032daa7770c44a8aa5d0171610287234 at {username='oryx', principal='oryx/127.5.22.60@KRBTEST.COM'} at 127.0.0.1:48236: session id = 032daa7770c44a8aa5d0171610287234-00000000000000000000000000000000
I20260928 06:35:42.889845 11399 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 06:35:42.891239 11951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 06:35:42.893839 11951 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234: tablet copy: Starting download of 0 data blocks...
I20260928 06:35:42.894034 11951 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234: tablet copy: Starting download of 1 WAL segments...
I20260928 06:35:42.895655 11951 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 06:35:42.897559 11399 tablet_copy_service.cc:342] P 5973a583576742e5a79e1b84b9e53783: Request end of tablet copy session 032daa7770c44a8aa5d0171610287234-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.22.60@KRBTEST.COM'} at 127.0.0.1:48236
I20260928 06:35:42.897648 11399 tablet_copy_service.cc:433] P 5973a583576742e5a79e1b84b9e53783: ending tablet copy session 032daa7770c44a8aa5d0171610287234-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 032daa7770c44a8aa5d0171610287234
I20260928 06:35:42.905105 11951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:42.906978 12142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:42.907377 12145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:42.910832 12143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:42.911240 11951 server_base.cc:1034] running on GCE node
I20260928 06:35:42.911402 11951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:42.911710 11951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:42.912842 11951 hybrid_clock.cc:648] HybridClock initialized: now 1790577342912827 us; error 33 us; skew 500 ppm
Sep 28 06:35:42 dist-test-slave-j0qf krb5kdc[11334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577342, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:42.915203 11951 init.cc:377] Logged in from keytab as oryx/127.5.22.60@KRBTEST.COM (short username oryx)
I20260928 06:35:42.915784 11951 webserver.cc:460] Webserver started at http://127.5.22.60:34403/ using document root <none> and password file <none>
I20260928 06:35:42.916194 11951 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:42.916301 11951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:42.917826 11951 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:42.919301 12152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:42.919607 11951 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:42.919728 11951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/wal
uuid: "032daa7770c44a8aa5d0171610287234"
format_stamp: "Formatted at 2026-09-28 06:35:42 on dist-test-slave-j0qf"
server_key: "b03f607f0d25627615779c7ef5fad37a"
server_key_iv: "aed77a5d33bb86a30c59a008a3ceae95"
server_key_version: "encryptionkey@0"
I20260928 06:35:42.919947 11951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:42.931372 11951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:42.934046 11951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:42.939926 11951 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.60:46045
I20260928 06:35:42.940263 12204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.60:46045 every 8 connection(s)
I20260928 06:35:42.940768 11951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/master-2/data/info.pb
I20260928 06:35:42.942975 12205 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:35:42.944001 12205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234: Bootstrap starting.
I20260928 06:35:42.950006 11704 heartbeater.cc:503] Master 127.5.22.62:38463 was elected leader, sending a full tablet report...
I20260928 06:35:42.951894 12205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:35:42.952174 12205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234: Bootstrap complete.
W20260928 06:35:42.952061 12123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:42.952414 12123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:42.952548 12123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 06:35:42.952811 12205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: NON_VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: true } }
I20260928 06:35:42.952924 12205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 032daa7770c44a8aa5d0171610287234, State: Initialized, Role: LEARNER
I20260928 06:35:42.953039 12205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: NON_VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: true } }
I20260928 06:35:42.953473 12205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:42.953399 12211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: NON_VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: true } } }
I20260928 06:35:42.953567 12211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [sys.catalog]: This master's current role is: LEARNER
I20260928 06:35:42.956185 12222 catalog_manager.cc:1259] Loaded cluster ID: 6c9264db8f9d43f884348bed61629dcc
I20260928 06:35:42.956307 12222 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234: loading cluster ID for follower catalog manager: success
I20260928 06:35:42.957993 12222 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234: acquiring CA information for follower catalog manager: success
W20260928 06:35:42.958302 12123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:42.958405 12123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:42.958549 12123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:35:42.959226 12222 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:35:42.964637 12123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.1.keytab
--principal=oryx/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:44565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=39681
--tserver_master_addrs=127.5.22.62:38463,127.5.22.61:39427,127.5.22.60:46045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:42.965775 12123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:42.967160 12123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:42.979357 12224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:42.979346 12225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:42.979342 12227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:42.979611 12123 server_base.cc:1034] running on GCE node
I20260928 06:35:42.980058 12123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:42.980695 12123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:42.981858 12123 hybrid_clock.cc:648] HybridClock initialized: now 1790577342981841 us; error 25 us; skew 500 ppm
Sep 28 06:35:42 dist-test-slave-j0qf krb5kdc[11334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577342, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:42.984614 12123 init.cc:377] Logged in from keytab as oryx/127.5.22.1@KRBTEST.COM (short username oryx)
I20260928 06:35:42.985493 12123 webserver.cc:460] Webserver started at http://127.5.22.1:39681/ using document root <none> and password file <none>
I20260928 06:35:42.986076 12123 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:42.986219 12123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:42.990232 12123 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:42.992482 12234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:42.993742 12123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 06:35:42.993875 12123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "50444827a7fb4637bee08e5afafa8f84"
format_stamp: "Formatted at 2026-09-28 06:35:41 on dist-test-slave-j0qf"
server_key: "eaef8d1e576462505b725e1b172288d2"
server_key_iv: "f66b967acdd970dace87d8791946b495"
server_key_version: "encryptionkey@0"
I20260928 06:35:42.994560 12123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:43.010875 12123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:43.014266 12123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:43.014514 12123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:43.015273 12123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:43.016228 12123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:43.016296 12123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:43.016364 12123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:43.016394 12123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:43.025650 12123 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:44565
I20260928 06:35:43.025673 12347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:44565 every 8 connection(s)
I20260928 06:35:43.026800 12123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:43.027241 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 12123
I20260928 06:35:43.027460 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 11575
I20260928 06:35:43.036163 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:41085
--local_ip_for_outbound_sockets=127.5.22.2
--tserver_master_addrs=127.5.22.62:38463
--webserver_port=38429
--webserver_interface=127.5.22.2
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.2.keytab
--principal=oryx/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.22.62:38463,127.5.22.61:39427
--tserver_master_addrs=127.5.22.62:38463,127.5.22.61:39427,127.5.22.60:46045 with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:35:43 dist-test-slave-j0qf krb5kdc[11334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577342, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.61@KRBTEST.COM
Sep 28 06:35:43 dist-test-slave-j0qf krb5kdc[11334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577342, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.60@KRBTEST.COM
Sep 28 06:35:43 dist-test-slave-j0qf krb5kdc[11334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577342, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM
I20260928 06:35:43.046066 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.030584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:32955 (local address 127.5.22.61:39427)
0928 06:35:43.030750 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:43.030754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:43.030772 (+ 18us) server_negotiation.cc:407] Connection header received
0928 06:35:43.031736 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:43.031739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:43.031795 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:43.031848 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:43.033909 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.034803 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.037007 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.037338 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.037418 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.040989 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:43.040993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:43.040996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:43.041024 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:43.042846 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.043541 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.043547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.043549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.043648 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.044092 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.044097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.044100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.044370 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:43.044498 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:43.045681 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0928 06:35:43.045830 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 06:35:43.047660 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.030929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:35275 (local address 127.5.22.62:38463)
0928 06:35:43.031061 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:43.031065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:43.031301 (+ 236us) server_negotiation.cc:407] Connection header received
0928 06:35:43.031736 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:43.031739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:43.031795 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:43.031848 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:43.033909 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.034808 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.036897 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.037227 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.037301 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.043012 (+ 5711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:43.043016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:43.043018 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:43.043051 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:43.045332 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.046007 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.046010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.046012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.046132 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.046642 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.046645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.046646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.046847 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:43.046923 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:43.047390 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 06:35:43.047509 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:35:43.047690 12353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.030153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:39427 (local address 127.5.22.1:32955)
0928 06:35:43.030615 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:43.030629 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:43.031497 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:43.032177 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:43.032186 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:43.032856 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:43.033740 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.033759 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.034991 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.034994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.036894 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.036902 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.037557 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.037559 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.037803 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.038701 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:43.038723 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:43.040777 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:43.042963 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:43.042972 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:43.042993 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:43.043384 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:43.043762 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:43.043767 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:43.043769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:43.043975 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:43.044565 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:43.044574 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:43.044944 (+ 370us) client_negotiation.cc:769] Sending connection context
0928 06:35:43.045196 (+ 252us) client_negotiation.cc:240] Negotiation successful
0928 06:35:43.046371 (+ 1175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":775,"spinlock_wait_cycles":3328,"thread_start_us":105,"threads_started":1}
I20260928 06:35:43.048593 12355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.030781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:53163 (local address 127.5.22.60:46045)
0928 06:35:43.031033 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:43.031038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:43.031067 (+ 29us) server_negotiation.cc:407] Connection header received
0928 06:35:43.031830 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:43.031837 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:43.031902 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:43.031985 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:43.033906 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.035113 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.036894 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.037304 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.037430 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.043189 (+ 5759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:43.043196 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:43.043201 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:43.043234 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:43.045417 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.046007 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.046011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.046013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.046270 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.046628 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.046631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.046633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.046847 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:43.046941 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:43.048300 (+ 1359us) server_negotiation.cc:299] Negotiation successful
0928 06:35:43.048436 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":86,"threads_started":1}
I20260928 06:35:43.048913 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.029365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:46045 (local address 127.5.22.1:53163)
0928 06:35:43.030155 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:43.030218 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:43.031491 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:43.032177 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:43.032188 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:43.032861 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:43.033739 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.033760 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.035253 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.035259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.036751 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.036765 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.037587 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.037593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.037775 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.038724 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:43.038742 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:43.043018 (+ 4276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:43.045524 (+ 2506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:43.045529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:43.045533 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:43.045890 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:43.046398 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:43.046403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:43.046405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:43.046513 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:43.046988 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:43.046991 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:43.047173 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 06:35:43.047303 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 06:35:43.047441 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":305,"spinlock_wait_cycles":55808,"thread_start_us":169,"threads_started":1}
I20260928 06:35:43.049144 12350 heartbeater.cc:347] Connected to a master server at 127.5.22.61:39427
I20260928 06:35:43.049376 12350 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:43.049674 12354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.030497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.5.22.1:35275)
0928 06:35:43.031206 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:43.031217 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:43.031497 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:43.032177 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:43.032186 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:43.032856 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:43.033739 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.033759 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.034991 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.034994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.036745 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.036758 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.037513 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.037519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.037803 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.038837 (+ 1034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:43.038855 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:43.042657 (+ 3802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:43.045466 (+ 2809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:43.045470 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:43.045475 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:43.045890 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:43.046413 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:43.046424 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:43.046425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:43.046518 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:43.046992 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:43.046994 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:43.047173 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 06:35:43.049326 (+ 2153us) client_negotiation.cc:240] Negotiation successful
0928 06:35:43.049509 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":520,"spinlock_wait_cycles":58240,"thread_start_us":69,"threads_started":1}
I20260928 06:35:43.050055 12350 heartbeater.cc:511] Master 127.5.22.61:39427 requested a full tablet report, sending...
I20260928 06:35:43.050663 12349 heartbeater.cc:347] Connected to a master server at 127.5.22.62:38463
I20260928 06:35:43.050762 12349 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:43.050936 12349 heartbeater.cc:511] Master 127.5.22.62:38463 requested a full tablet report, sending...
I20260928 06:35:43.051162 12348 heartbeater.cc:347] Connected to a master server at 127.5.22.60:46045
I20260928 06:35:43.051229 12348 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:43.051376 12348 heartbeater.cc:511] Master 127.5.22.60:46045 requested a full tablet report, sending...
I20260928 06:35:43.051766 11376 ts_manager.cc:194] Re-registered known tserver with Master: 50444827a7fb4637bee08e5afafa8f84 (127.5.22.1:44565)
I20260928 06:35:43.051963 11979 ts_manager.cc:194] Registered new tserver with Master: 50444827a7fb4637bee08e5afafa8f84 (127.5.22.1:44565)
I20260928 06:35:43.052500 11376 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.22.1@KRBTEST.COM'} at 127.5.22.1:35275
I20260928 06:35:43.055164 12169 ts_manager.cc:194] Registered new tserver with Master: 50444827a7fb4637bee08e5afafa8f84 (127.5.22.1:44565)
W20260928 06:35:43.155074 12356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:43.155375 12356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:43.155429 12356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:43.159468 12356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:43.159529 12356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:43.159591 12356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:35:43.164979 12356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.2.keytab
--principal=oryx/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:41085
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=38429
--tserver_master_addrs=127.5.22.62:38463,127.5.22.61:39427,127.5.22.60:46045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:43.166072 12356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:43.167115 12356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:35:43.170754 11838 heartbeater.cc:503] Master 127.5.22.62:38463 was elected leader, sending a full tablet report...
W20260928 06:35:43.178095 12361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:43.178088 12362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:43.178088 12364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:43.178367 12356 server_base.cc:1034] running on GCE node
I20260928 06:35:43.179104 12356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:43.179864 12356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:43.181087 12356 hybrid_clock.cc:648] HybridClock initialized: now 1790577343181047 us; error 51 us; skew 500 ppm
Sep 28 06:35:43 dist-test-slave-j0qf krb5kdc[11334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577343, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:43.184221 12356 init.cc:377] Logged in from keytab as oryx/127.5.22.2@KRBTEST.COM (short username oryx)
I20260928 06:35:43.185359 12356 webserver.cc:460] Webserver started at http://127.5.22.2:38429/ using document root <none> and password file <none>
I20260928 06:35:43.185943 12356 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:43.186017 12356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:43.190044 12356 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:43.192479 12371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:43.193871 12356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 06:35:43.194028 12356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "b63caecbb37e43f58339a2f1a616de2c"
format_stamp: "Formatted at 2026-09-28 06:35:41 on dist-test-slave-j0qf"
server_key: "a0e055edc6a82e03b0f41313470766a5"
server_key_iv: "cd9d931c4c70c38ecb9404ab35fd34f1"
server_key_version: "encryptionkey@0"
I20260928 06:35:43.194679 12356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:43.226928 12356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:43.230535 12356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:43.230756 12356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:43.231489 12356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:43.232434 12356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:43.232482 12356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:43.232524 12356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:43.232583 12356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:43.242274 12356 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:41085
I20260928 06:35:43.242292 12484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:41085 every 8 connection(s)
I20260928 06:35:43.243377 12356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:43.246165 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 12356
I20260928 06:35:43.246369 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 11709
I20260928 06:35:43.255285 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:37615
--local_ip_for_outbound_sockets=127.5.22.3
--tserver_master_addrs=127.5.22.62:38463
--webserver_port=45357
--webserver_interface=127.5.22.3
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.3.keytab
--principal=oryx/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.22.62:38463,127.5.22.61:39427
--tserver_master_addrs=127.5.22.62:38463,127.5.22.61:39427,127.5.22.60:46045 with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:35:43 dist-test-slave-j0qf krb5kdc[11334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577343, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.60@KRBTEST.COM
Sep 28 06:35:43 dist-test-slave-j0qf krb5kdc[11334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577343, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM
Sep 28 06:35:43 dist-test-slave-j0qf krb5kdc[11334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577343, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.61@KRBTEST.COM
I20260928 06:35:43.262663 12355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.248503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:40197 (local address 127.5.22.60:46045)
0928 06:35:43.249163 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:43.249170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:43.249187 (+ 17us) server_negotiation.cc:407] Connection header received
0928 06:35:43.249233 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:43.249239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:43.249295 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:43.249378 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:43.250054 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.250995 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.252790 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.253265 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.253378 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.258147 (+ 4769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:43.258152 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:43.258160 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:43.258188 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:43.260252 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.260840 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.260843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.260845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.260923 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.261242 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.261245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.261247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.261477 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:43.261560 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:43.262352 (+ 792us) server_negotiation.cc:299] Negotiation successful
0928 06:35:43.262477 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556}
I20260928 06:35:43.262663 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.245274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:44655 (local address 127.5.22.61:39427)
0928 06:35:43.245447 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:43.245454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:43.247776 (+ 2322us) server_negotiation.cc:407] Connection header received
0928 06:35:43.247827 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:43.247833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:43.247880 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:43.247954 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:43.249762 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.250614 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.252836 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.253227 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.253313 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.258840 (+ 5527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:43.258844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:43.258847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:43.258880 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:43.260691 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.261176 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.261182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.261185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.261277 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.261654 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.261657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.261659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.261877 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:43.261944 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:43.262352 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 06:35:43.262477 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 06:35:43.262811 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.245304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:43731 (local address 127.5.22.62:38463)
0928 06:35:43.245433 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:43.245437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:43.247176 (+ 1739us) server_negotiation.cc:407] Connection header received
0928 06:35:43.247716 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:43.247722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:43.247788 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:43.247844 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:43.249762 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.250614 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.252836 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.253227 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.253313 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.257733 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:43.257738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:43.257741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:43.257775 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:43.259784 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.260524 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.260531 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.260534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.260644 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.261133 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.261139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.261144 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.261430 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:43.261520 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:43.262436 (+ 916us) server_negotiation.cc:299] Negotiation successful
0928 06:35:43.262586 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 06:35:43.264364 12491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.245707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.5.22.2:43731)
0928 06:35:43.247076 (+ 1369us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:43.247092 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:43.247477 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:43.248183 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:43.248192 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:43.248847 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:43.249584 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.249603 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.250799 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.250802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.252660 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.252675 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.253548 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.253552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.253901 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.254892 (+ 991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:43.254915 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:43.257524 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:43.259931 (+ 2407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:43.259940 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:43.259961 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:43.260370 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:43.260782 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:43.260787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:43.260790 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:43.261002 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:43.261620 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:43.261630 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:43.262059 (+ 429us) client_negotiation.cc:769] Sending connection context
0928 06:35:43.263533 (+ 1474us) client_negotiation.cc:240] Negotiation successful
0928 06:35:43.263717 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1269,"mutex_wait_us":416,"spinlock_wait_cycles":82560,"thread_start_us":178,"threads_started":1}
I20260928 06:35:43.264621 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.246790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:46045 (local address 127.5.22.2:40197)
0928 06:35:43.247891 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:43.247907 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:43.247991 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:43.249520 (+ 1529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:43.249526 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:43.249744 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:43.249896 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.249906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.251155 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.251161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.252651 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.252664 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.253584 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.253590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.253915 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.255075 (+ 1160us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:43.255094 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:43.258004 (+ 2910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:43.260435 (+ 2431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:43.260440 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:43.260442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:43.260724 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:43.261044 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:43.261046 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:43.261048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:43.261139 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:43.261632 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:43.261637 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:43.262044 (+ 407us) client_negotiation.cc:769] Sending connection context
0928 06:35:43.262827 (+ 783us) client_negotiation.cc:240] Negotiation successful
0928 06:35:43.263196 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"mutex_wait_us":301,"spinlock_wait_cycles":28928,"thread_start_us":106,"threads_started":1}
I20260928 06:35:43.266124 12486 heartbeater.cc:347] Connected to a master server at 127.5.22.62:38463
I20260928 06:35:43.266177 12485 heartbeater.cc:347] Connected to a master server at 127.5.22.60:46045
I20260928 06:35:43.266402 12485 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:43.266986 12485 heartbeater.cc:511] Master 127.5.22.60:46045 requested a full tablet report, sending...
I20260928 06:35:43.266405 12486 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:43.267699 12486 heartbeater.cc:511] Master 127.5.22.62:38463 requested a full tablet report, sending...
I20260928 06:35:43.268066 11936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.263460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:46045 (local address 127.0.0.1:39354)
0928 06:35:43.263688 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:43.263703 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:43.263775 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:43.264309 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:43.264312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:43.264329 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:43.264430 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.264435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.265409 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.265413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.266558 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.266568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.267627 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.267630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.267726 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.267749 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 06:35:43.267843 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 06:35:43.267931 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260928 06:35:43.268349 12169 ts_manager.cc:194] Registered new tserver with Master: b63caecbb37e43f58339a2f1a616de2c (127.5.22.2:41085)
I20260928 06:35:43.268384 12355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.263905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39354 (local address 127.5.22.60:46045)
0928 06:35:43.264020 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:43.264024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:43.264036 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:35:43.264072 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:43.264075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:43.264125 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:43.264211 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:43.264567 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.265293 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.266695 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.267310 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.267413 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.267563 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:43.268219 (+ 656us) server_negotiation.cc:299] Negotiation successful
0928 06:35:43.268287 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:35:43.268728 11376 ts_manager.cc:194] Re-registered known tserver with Master: b63caecbb37e43f58339a2f1a616de2c (127.5.22.2:41085)
I20260928 06:35:43.268821 12179 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 06:35:43.269197 12211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5973a583576742e5a79e1b84b9e53783. Latest consensus state: current_term: 1 leader_uuid: "5973a583576742e5a79e1b84b9e53783" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: NON_VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: true } } }
I20260928 06:35:43.269304 12211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [sys.catalog]: This master's current role is: LEARNER
I20260928 06:35:43.269433 11376 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.22.2@KRBTEST.COM'} at 127.5.22.2:43731
I20260928 06:35:43.270184 12489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.245707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:39427 (local address 127.5.22.2:44655)
0928 06:35:43.246245 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:43.246297 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:43.247467 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:43.248183 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:43.248192 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:43.249421 (+ 1229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:43.249584 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.249603 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.250799 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.250802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.252684 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.252691 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.253548 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.253552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.253942 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.254952 (+ 1010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:43.254969 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:43.258659 (+ 3690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:43.260815 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:43.260821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:43.260823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:43.261067 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:43.261458 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:43.261461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:43.261464 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:43.261548 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:43.262027 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:43.262030 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:43.262111 (+ 81us) client_negotiation.cc:769] Sending connection context
0928 06:35:43.269843 (+ 7732us) client_negotiation.cc:240] Negotiation successful
0928 06:35:43.270003 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":425,"spinlock_wait_cycles":2432,"thread_start_us":175,"threads_started":1}
I20260928 06:35:43.270855 12487 heartbeater.cc:347] Connected to a master server at 127.5.22.61:39427
I20260928 06:35:43.270936 12487 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:43.271068 12487 heartbeater.cc:511] Master 127.5.22.61:39427 requested a full tablet report, sending...
I20260928 06:35:43.271679 11979 ts_manager.cc:194] Registered new tserver with Master: b63caecbb37e43f58339a2f1a616de2c (127.5.22.2:41085)
W20260928 06:35:43.373785 12492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:43.374079 12492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:43.374163 12492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:43.378136 12492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:43.378221 12492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:43.378314 12492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:35:43.383826 12492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.3.keytab
--principal=oryx/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:37615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=45357
--tserver_master_addrs=127.5.22.62:38463,127.5.22.61:39427,127.5.22.60:46045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:43.384991 12492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:43.386037 12492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:43.395802 12500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:43.395783 12497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:43.396034 12492 server_base.cc:1034] running on GCE node
W20260928 06:35:43.395788 12498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:43.396643 12492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:43.397233 12492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:43.398389 12492 hybrid_clock.cc:648] HybridClock initialized: now 1790577343398351 us; error 51 us; skew 500 ppm
Sep 28 06:35:43 dist-test-slave-j0qf krb5kdc[11334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577343, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:43.401245 12492 init.cc:377] Logged in from keytab as oryx/127.5.22.3@KRBTEST.COM (short username oryx)
I20260928 06:35:43.402062 12492 webserver.cc:460] Webserver started at http://127.5.22.3:45357/ using document root <none> and password file <none>
I20260928 06:35:43.402657 12492 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:43.402730 12492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:43.406591 12492 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260928 06:35:43.408553 12507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:43.409754 12492 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:43.409845 12492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "af81bb4b529c4a58824980974cb7362e"
format_stamp: "Formatted at 2026-09-28 06:35:42 on dist-test-slave-j0qf"
server_key: "6fff47fb134f2585aaa9cd0de7ee8f5b"
server_key_iv: "deba341bacc80f96857f71d0fbb625c9"
server_key_version: "encryptionkey@0"
I20260928 06:35:43.410439 12492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:43.424746 12492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:43.427922 12492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:43.428150 12492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:43.428743 12492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:43.429610 12492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:43.429680 12492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:43.429750 12492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:43.429801 12492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:43.438717 12492 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:37615
I20260928 06:35:43.438753 12620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:37615 every 8 connection(s)
I20260928 06:35:43.439733 12492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:43.443336 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 12492
Sep 28 06:35:43 dist-test-slave-j0qf krb5kdc[11334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577343, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.60@KRBTEST.COM
Sep 28 06:35:43 dist-test-slave-j0qf krb5kdc[11334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577343, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.61@KRBTEST.COM
I20260928 06:35:43.456591 12632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.445399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36482 (local address 127.5.22.61:39427)
0928 06:35:43.445691 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:43.445718 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:43.446066 (+ 348us) server_negotiation.cc:407] Connection header received
0928 06:35:43.446105 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:43.446108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:43.446151 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:43.446224 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:43.447276 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.448042 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.449220 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.449588 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.449689 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.451516 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:43.451522 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:43.451530 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:43.451550 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:43.454066 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.454848 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.454853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.454858 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.454940 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.455337 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.455340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.455342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.455570 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:43.455687 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:43.456291 (+ 604us) server_negotiation.cc:299] Negotiation successful
0928 06:35:43.456415 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"spinlock_wait_cycles":3840,"thread_start_us":69,"threads_started":1}
I20260928 06:35:43.456828 12355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.443132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:48651 (local address 127.5.22.60:46045)
0928 06:35:43.443270 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:43.443274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:43.443542 (+ 268us) server_negotiation.cc:407] Connection header received
0928 06:35:43.443857 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:43.443862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:43.443922 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:43.443978 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:43.445962 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.446801 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.448427 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.448764 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.448854 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.452589 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:43.452592 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:43.452595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:43.452620 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:43.454490 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.455091 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.455095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.455097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.455173 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.455562 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.455565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.455566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.455759 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:43.455839 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:43.456561 (+ 722us) server_negotiation.cc:299] Negotiation successful
0928 06:35:43.456673 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 06:35:43.457445 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.441588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:40207 (local address 127.5.22.61:39427)
0928 06:35:43.441728 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:43.441733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:43.442640 (+ 907us) server_negotiation.cc:407] Connection header received
0928 06:35:43.443939 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:43.443944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:43.443996 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:43.444064 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:43.445766 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.446589 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.448421 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.448760 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.448848 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.453093 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:43.453099 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:43.453103 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:43.453131 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:43.454799 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.455478 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.455481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.455483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.455559 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.455871 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.455874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.455876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.456057 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:43.456146 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:43.457199 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0928 06:35:43.457300 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:35:43.458305 12627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.443015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:46045 (local address 127.5.22.3:48651)
0928 06:35:43.443451 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:43.443468 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:43.443625 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:43.444311 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:43.444320 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:43.445750 (+ 1430us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:43.445849 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.445854 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.446925 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.446929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.448294 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.448303 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.448997 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.449001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.449235 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.449954 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:43.449973 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:43.452405 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:43.454604 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:43.454610 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:43.454625 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:43.454976 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:43.455261 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:43.455264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:43.455266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:43.455465 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:43.455887 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:43.455893 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:43.456180 (+ 287us) client_negotiation.cc:769] Sending connection context
0928 06:35:43.456366 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 06:35:43.457708 (+ 1342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":1188,"thread_start_us":80,"threads_started":1}
I20260928 06:35:43.458747 12626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.441995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:39427 (local address 127.5.22.3:40207)
0928 06:35:43.442492 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:43.442545 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:43.443626 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:43.444308 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:43.444317 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:43.444983 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:43.445614 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.445630 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.446768 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.446774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.448296 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.448305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.448997 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.449001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.449240 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.450092 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:43.450113 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:43.452708 (+ 2595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:43.454925 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:43.454928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:43.454931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:43.455173 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:43.455678 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:43.455681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:43.455683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:43.455768 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:43.456185 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:43.456187 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:43.456286 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 06:35:43.456382 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 06:35:43.456731 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":111,"spinlock_wait_cycles":34688,"thread_start_us":136,"threads_started":1}
I20260928 06:35:43.459928 12621 heartbeater.cc:347] Connected to a master server at 127.5.22.60:46045
I20260928 06:35:43.460239 12621 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:43.460458 12623 heartbeater.cc:347] Connected to a master server at 127.5.22.61:39427
I20260928 06:35:43.460553 12623 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:43.460990 12623 heartbeater.cc:511] Master 127.5.22.61:39427 requested a full tablet report, sending...
I20260928 06:35:43.460991 12621 heartbeater.cc:511] Master 127.5.22.60:46045 requested a full tablet report, sending...
I20260928 06:35:43.462023 11979 ts_manager.cc:194] Registered new tserver with Master: af81bb4b529c4a58824980974cb7362e (127.5.22.3:37615)
I20260928 06:35:43.462072 12169 ts_manager.cc:194] Registered new tserver with Master: af81bb4b529c4a58824980974cb7362e (127.5.22.3:37615)
Sep 28 06:35:43 dist-test-slave-j0qf krb5kdc[11334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577343, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM
I20260928 06:35:43.467841 12355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.459113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39368 (local address 127.5.22.60:46045)
0928 06:35:43.459239 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:43.459242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:43.459833 (+ 591us) server_negotiation.cc:407] Connection header received
0928 06:35:43.460012 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:43.460015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:43.460063 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:43.460111 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:43.460959 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.461657 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.462721 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.462940 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.462993 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.464522 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:43.464526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:43.464528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:43.464551 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:43.466132 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.466616 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.466620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.466622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.466699 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.466969 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.466973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.466975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.467160 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:43.467320 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:43.467586 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 06:35:43.467689 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:35:43.469383 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.453169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48240 (local address 127.5.22.62:38463)
0928 06:35:43.457335 (+ 4166us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:43.457339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:43.459719 (+ 2380us) server_negotiation.cc:407] Connection header received
0928 06:35:43.459756 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:43.459758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:43.459805 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:43.459880 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:43.460791 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.461525 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.462621 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.462957 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.463067 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.464967 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:43.464974 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:43.464979 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:43.465013 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:43.467183 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.467690 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.467695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.467699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.467781 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.468072 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.468077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.468079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.468278 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:43.468412 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:43.469080 (+ 668us) server_negotiation.cc:299] Negotiation successful
0928 06:35:43.469200 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4088,"thread_start_us":59,"threads_started":1}
I20260928 06:35:43.469624 12625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.441994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.5.22.3:49757)
0928 06:35:43.442491 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:43.442543 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:43.443626 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:43.444308 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:43.444317 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:43.449624 (+ 5307us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:43.449737 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.449743 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.457917 (+ 8174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.457920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.458587 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.458597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.462762 (+ 4165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.462765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.462857 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.463382 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:43.463398 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:43.465490 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:43.467609 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:43.467615 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:43.467620 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:43.468216 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:43.468516 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:43.468522 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:43.468527 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:43.468657 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:43.469106 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:43.469109 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:43.469206 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 06:35:43.469305 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 06:35:43.469436 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"spinlock_wait_cycles":3456,"thread_start_us":144,"threads_started":1}
I20260928 06:35:43.469671 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.441501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:49757 (local address 127.5.22.62:38463)
0928 06:35:43.441771 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:43.441777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:43.442642 (+ 865us) server_negotiation.cc:407] Connection header received
0928 06:35:43.443862 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:43.443866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:43.443924 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:43.443977 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:43.457073 (+ 13096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.457747 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.459561 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.459898 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.459967 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.465639 (+ 5672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:43.465642 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:43.465645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:43.465669 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:43.467469 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.468323 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.468326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.468328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.468414 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.468750 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.468753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.468755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.468965 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:43.469060 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:43.469380 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 06:35:43.469526 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260928 06:35:43.470368 12622 heartbeater.cc:347] Connected to a master server at 127.5.22.62:38463
I20260928 06:35:43.470484 12622 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:43.470728 12622 heartbeater.cc:511] Master 127.5.22.62:38463 requested a full tablet report, sending...
I20260928 06:35:43.471396 11376 ts_manager.cc:194] Re-registered known tserver with Master: af81bb4b529c4a58824980974cb7362e (127.5.22.3:37615)
I20260928 06:35:43.471993 11376 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.22.3@KRBTEST.COM'} at 127.5.22.3:49757
I20260928 06:35:43.472644 11379 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:35:43.475710 11379 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:43.478152 12133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5973a583576742e5a79e1b84b9e53783" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: NON_VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: true } } }
I20260928 06:35:43.478299 12133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:43.480185 12640 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783: attempting to promote NON_VOTER 032daa7770c44a8aa5d0171610287234 to VOTER
I20260928 06:35:43.480516 12640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: false } }
I20260928 06:35:43.481930 11989 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [term 1 FOLLOWER]: Refusing update from remote peer 5973a583576742e5a79e1b84b9e53783: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 06:35:43.482028 12179 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [term 1 LEARNER]: Refusing update from remote peer 5973a583576742e5a79e1b84b9e53783: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 06:35:43.482443 12133 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 06:35:43.482630 12640 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 06:35:43.484342 12133 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 032daa7770c44a8aa5d0171610287234 (127.5.22.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: false } } }
I20260928 06:35:43.484858 11989 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 032daa7770c44a8aa5d0171610287234 (127.5.22.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: false } } }
I20260928 06:35:43.485836 12133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5973a583576742e5a79e1b84b9e53783" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: false } } }
I20260928 06:35:43.485934 12133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:43.486057 12133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5973a583576742e5a79e1b84b9e53783" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: false } } }
I20260928 06:35:43.486166 12133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:43.486966 12647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5973a583576742e5a79e1b84b9e53783" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: false } } }
I20260928 06:35:43.487100 12647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:35:43.488444 12639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5973a583576742e5a79e1b84b9e53783" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: false } } }
I20260928 06:35:43.488565 12639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:43.488978 12179 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 032daa7770c44a8aa5d0171610287234 (127.5.22.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: false } } }
I20260928 06:35:43.490512 12211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5973a583576742e5a79e1b84b9e53783" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: false } } }
I20260928 06:35:43.490629 12211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:35:43.499930 12649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.492914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:41085 (local address 127.0.0.1:45058)
0928 06:35:43.493146 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:43.493157 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:43.493257 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:43.494407 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:43.494440 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:43.494461 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:43.494628 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.494636 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.496409 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.496414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.498014 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.498029 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.499420 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.499430 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.499526 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.499551 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 06:35:43.499643 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 06:35:43.499723 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":152,"thread_start_us":105,"threads_started":1}
I20260928 06:35:43.499969 12652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.493638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.5.22.2:41085)
0928 06:35:43.493959 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:43.493971 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:43.493991 (+ 20us) server_negotiation.cc:407] Connection header received
0928 06:35:43.494059 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:43.494068 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:43.494238 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:43.494431 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:43.495137 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.496252 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.498186 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.499149 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.499270 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.499435 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:43.499692 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 06:35:43.499808 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":148,"threads_started":1}
I20260928 06:35:43.500327 11936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.492683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:37615 (local address 127.0.0.1:43886)
0928 06:35:43.492802 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:43.492821 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:43.492911 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:43.494011 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:43.494018 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:43.494039 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:43.494148 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.494153 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.496133 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.496137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.498091 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.498104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.499915 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.499919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.500022 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.500048 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 06:35:43.500131 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 06:35:43.500205 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":3}
I20260928 06:35:43.500495 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.493055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43886 (local address 127.5.22.3:37615)
0928 06:35:43.493429 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:43.493442 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:43.493467 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:35:43.493560 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:43.493567 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:43.493816 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:43.494320 (+ 504us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:43.494493 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.495905 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.498247 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.499332 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.499459 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.499611 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:43.500192 (+ 581us) server_negotiation.cc:299] Negotiation successful
0928 06:35:43.500310 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":113,"threads_started":1}
I20260928 06:35:43.504014 12555 tablet_service.cc:1467] Processing CreateTablet for tablet 6951ec8504424188944753df2101db70 (DEFAULT_TABLE table=test-table [id=a26b78d59dbf4ff2af9ca76b9c9b10b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:43.504029 12419 tablet_service.cc:1467] Processing CreateTablet for tablet 6951ec8504424188944753df2101db70 (DEFAULT_TABLE table=test-table [id=a26b78d59dbf4ff2af9ca76b9c9b10b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:43.504966 12555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6951ec8504424188944753df2101db70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:43.505105 12419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6951ec8504424188944753df2101db70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:43.506494 12651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.493257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:44565 (local address 127.0.0.1:34516)
0928 06:35:43.493467 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:43.493482 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:43.493561 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:43.496937 (+ 3376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:43.496941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:43.496962 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:43.497066 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.497073 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.502797 (+ 5724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.502801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.504089 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.504102 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.506024 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.506028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.506125 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.506151 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 06:35:43.506234 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 06:35:43.506309 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":81,"threads_started":1}
I20260928 06:35:43.507148 12653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.493778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34516 (local address 127.5.22.1:44565)
0928 06:35:43.496271 (+ 2493us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:43.496290 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:43.496312 (+ 22us) server_negotiation.cc:407] Connection header received
0928 06:35:43.496390 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:43.496394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:43.496591 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:43.496763 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:43.497213 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.502642 (+ 5429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.504253 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.505673 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.505791 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.505950 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:43.506881 (+ 931us) server_negotiation.cc:299] Negotiation successful
0928 06:35:43.506993 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2383,"thread_start_us":80,"threads_started":1}
I20260928 06:35:43.510092 12282 tablet_service.cc:1467] Processing CreateTablet for tablet 6951ec8504424188944753df2101db70 (DEFAULT_TABLE table=test-table [id=a26b78d59dbf4ff2af9ca76b9c9b10b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:43.511111 12282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6951ec8504424188944753df2101db70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:43.512573 12654 tablet_bootstrap.cc:492] T 6951ec8504424188944753df2101db70 P b63caecbb37e43f58339a2f1a616de2c: Bootstrap starting.
I20260928 06:35:43.514972 12655 tablet_bootstrap.cc:492] T 6951ec8504424188944753df2101db70 P af81bb4b529c4a58824980974cb7362e: Bootstrap starting.
I20260928 06:35:43.515516 12654 tablet_bootstrap.cc:654] T 6951ec8504424188944753df2101db70 P b63caecbb37e43f58339a2f1a616de2c: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:43.516481 12654 log.cc:824] T 6951ec8504424188944753df2101db70 P b63caecbb37e43f58339a2f1a616de2c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:43.517169 12655 tablet_bootstrap.cc:654] T 6951ec8504424188944753df2101db70 P af81bb4b529c4a58824980974cb7362e: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:43.517964 12655 log.cc:824] T 6951ec8504424188944753df2101db70 P af81bb4b529c4a58824980974cb7362e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:43.518266 12657 tablet_bootstrap.cc:492] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84: Bootstrap starting.
I20260928 06:35:43.518746 12654 tablet_bootstrap.cc:492] T 6951ec8504424188944753df2101db70 P b63caecbb37e43f58339a2f1a616de2c: No bootstrap required, opened a new log
I20260928 06:35:43.518952 12654 ts_tablet_manager.cc:1397] T 6951ec8504424188944753df2101db70 P b63caecbb37e43f58339a2f1a616de2c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 06:35:43.519902 12655 tablet_bootstrap.cc:492] T 6951ec8504424188944753df2101db70 P af81bb4b529c4a58824980974cb7362e: No bootstrap required, opened a new log
I20260928 06:35:43.520072 12655 ts_tablet_manager.cc:1397] T 6951ec8504424188944753df2101db70 P af81bb4b529c4a58824980974cb7362e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 06:35:43.520491 12657 tablet_bootstrap.cc:654] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:43.521168 12657 log.cc:824] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:43.522832 12657 tablet_bootstrap.cc:492] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84: No bootstrap required, opened a new log
I20260928 06:35:43.523015 12657 ts_tablet_manager.cc:1397] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 06:35:43.523607 12654 raft_consensus.cc:348] T 6951ec8504424188944753df2101db70 P b63caecbb37e43f58339a2f1a616de2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63caecbb37e43f58339a2f1a616de2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41085 } } peers { permanent_uuid: "af81bb4b529c4a58824980974cb7362e" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 37615 } } peers { permanent_uuid: "50444827a7fb4637bee08e5afafa8f84" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44565 } }
I20260928 06:35:43.523869 12654 raft_consensus.cc:374] T 6951ec8504424188944753df2101db70 P b63caecbb37e43f58339a2f1a616de2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:43.523924 12654 raft_consensus.cc:729] T 6951ec8504424188944753df2101db70 P b63caecbb37e43f58339a2f1a616de2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63caecbb37e43f58339a2f1a616de2c, State: Initialized, Role: FOLLOWER
I20260928 06:35:43.524348 12654 consensus_queue.cc:260] T 6951ec8504424188944753df2101db70 P b63caecbb37e43f58339a2f1a616de2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63caecbb37e43f58339a2f1a616de2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41085 } } peers { permanent_uuid: "af81bb4b529c4a58824980974cb7362e" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 37615 } } peers { permanent_uuid: "50444827a7fb4637bee08e5afafa8f84" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44565 } }
I20260928 06:35:43.524542 12655 raft_consensus.cc:348] T 6951ec8504424188944753df2101db70 P af81bb4b529c4a58824980974cb7362e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63caecbb37e43f58339a2f1a616de2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41085 } } peers { permanent_uuid: "af81bb4b529c4a58824980974cb7362e" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 37615 } } peers { permanent_uuid: "50444827a7fb4637bee08e5afafa8f84" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44565 } }
I20260928 06:35:43.524775 12655 raft_consensus.cc:374] T 6951ec8504424188944753df2101db70 P af81bb4b529c4a58824980974cb7362e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:43.524820 12655 raft_consensus.cc:729] T 6951ec8504424188944753df2101db70 P af81bb4b529c4a58824980974cb7362e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af81bb4b529c4a58824980974cb7362e, State: Initialized, Role: FOLLOWER
I20260928 06:35:43.525210 12655 consensus_queue.cc:260] T 6951ec8504424188944753df2101db70 P af81bb4b529c4a58824980974cb7362e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63caecbb37e43f58339a2f1a616de2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41085 } } peers { permanent_uuid: "af81bb4b529c4a58824980974cb7362e" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 37615 } } peers { permanent_uuid: "50444827a7fb4637bee08e5afafa8f84" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44565 } }
I20260928 06:35:43.525346 12654 ts_tablet_manager.cc:1428] T 6951ec8504424188944753df2101db70 P b63caecbb37e43f58339a2f1a616de2c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 06:35:43.526352 12655 ts_tablet_manager.cc:1428] T 6951ec8504424188944753df2101db70 P af81bb4b529c4a58824980974cb7362e: Time spent starting tablet: real 0.006s user 0.001s sys 0.007s
I20260928 06:35:43.526619 12486 heartbeater.cc:503] Master 127.5.22.62:38463 was elected leader, sending a full tablet report...
I20260928 06:35:43.526664 12622 heartbeater.cc:503] Master 127.5.22.62:38463 was elected leader, sending a full tablet report...
I20260928 06:35:43.527283 12657 raft_consensus.cc:348] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63caecbb37e43f58339a2f1a616de2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41085 } } peers { permanent_uuid: "af81bb4b529c4a58824980974cb7362e" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 37615 } } peers { permanent_uuid: "50444827a7fb4637bee08e5afafa8f84" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44565 } }
I20260928 06:35:43.529758 12657 raft_consensus.cc:374] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:43.529835 12657 raft_consensus.cc:729] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50444827a7fb4637bee08e5afafa8f84, State: Initialized, Role: FOLLOWER
I20260928 06:35:43.530254 12657 consensus_queue.cc:260] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63caecbb37e43f58339a2f1a616de2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41085 } } peers { permanent_uuid: "af81bb4b529c4a58824980974cb7362e" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 37615 } } peers { permanent_uuid: "50444827a7fb4637bee08e5afafa8f84" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44565 } }
I20260928 06:35:43.531442 12657 ts_tablet_manager.cc:1428] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84: Time spent starting tablet: real 0.008s user 0.003s sys 0.002s
I20260928 06:35:43.531581 12349 heartbeater.cc:503] Master 127.5.22.62:38463 was elected leader, sending a full tablet report...
W20260928 06:35:43.691565 12624 tablet.cc:2369] T 6951ec8504424188944753df2101db70 P af81bb4b529c4a58824980974cb7362e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:35:43.744975 12488 tablet.cc:2369] T 6951ec8504424188944753df2101db70 P b63caecbb37e43f58339a2f1a616de2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:35:43.749994 12662 raft_consensus.cc:482] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:43.750250 12662 raft_consensus.cc:504] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63caecbb37e43f58339a2f1a616de2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41085 } } peers { permanent_uuid: "af81bb4b529c4a58824980974cb7362e" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 37615 } } peers { permanent_uuid: "50444827a7fb4637bee08e5afafa8f84" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44565 } }
I20260928 06:35:43.751606 12662 leader_election.cc:288] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b63caecbb37e43f58339a2f1a616de2c (127.5.22.2:41085), af81bb4b529c4a58824980974cb7362e (127.5.22.3:37615)
I20260928 06:35:43.756249 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.752047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:56163 (local address 127.5.22.3:37615)
0928 06:35:43.752171 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:43.752176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:43.752306 (+ 130us) server_negotiation.cc:407] Connection header received
0928 06:35:43.752563 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:43.752567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:43.752618 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:43.752703 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:43.752972 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.753544 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.755349 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.755747 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.755835 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.755894 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:43.756065 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 06:35:43.756136 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:35:43.756273 12663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.751895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:37615 (local address 127.5.22.1:56163)
0928 06:35:43.752220 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:43.752232 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:43.752380 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:43.752721 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:43.752724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:43.752740 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:43.752866 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.752871 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.753693 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.753699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.755215 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.755222 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.755852 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.755855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.755930 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.755949 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 06:35:43.756043 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 06:35:43.756116 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":142,"spinlock_wait_cycles":8448,"thread_start_us":83,"threads_started":1}
I20260928 06:35:43.756950 12664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.751827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:41085 (local address 127.5.22.1:50833)
0928 06:35:43.752202 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:43.752215 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:43.752384 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:43.752667 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:43.752670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:43.752687 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:43.752903 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.752911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.753741 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.753744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.755212 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:43.755223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.756575 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.756580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:43.756679 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.756706 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 06:35:43.756785 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:35:43.756848 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":134,"spinlock_wait_cycles":4352,"thread_start_us":95,"threads_started":1}
I20260928 06:35:43.757021 12652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.751958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:50833 (local address 127.5.22.2:41085)
0928 06:35:43.752101 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:43.752108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:43.752296 (+ 188us) server_negotiation.cc:407] Connection header received
0928 06:35:43.752500 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:43.752503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:43.752558 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:43.752645 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:43.753028 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.753623 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.755943 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.756444 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.756548 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.756600 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:43.756817 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 06:35:43.756910 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 06:35:43.757042 12575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6951ec8504424188944753df2101db70" candidate_uuid: "50444827a7fb4637bee08e5afafa8f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af81bb4b529c4a58824980974cb7362e" is_pre_election: true
I20260928 06:35:43.757356 12575 raft_consensus.cc:2454] T 6951ec8504424188944753df2101db70 P af81bb4b529c4a58824980974cb7362e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50444827a7fb4637bee08e5afafa8f84 in term 0.
I20260928 06:35:43.757912 12238 leader_election.cc:302] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50444827a7fb4637bee08e5afafa8f84, af81bb4b529c4a58824980974cb7362e; no voters:
I20260928 06:35:43.758191 12662 raft_consensus.cc:2781] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:43.758217 12439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6951ec8504424188944753df2101db70" candidate_uuid: "50444827a7fb4637bee08e5afafa8f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63caecbb37e43f58339a2f1a616de2c" is_pre_election: true
I20260928 06:35:43.758287 12662 raft_consensus.cc:482] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:43.758364 12662 raft_consensus.cc:3037] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:43.758535 12439 raft_consensus.cc:2454] T 6951ec8504424188944753df2101db70 P b63caecbb37e43f58339a2f1a616de2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50444827a7fb4637bee08e5afafa8f84 in term 0.
I20260928 06:35:43.759541 12662 raft_consensus.cc:504] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63caecbb37e43f58339a2f1a616de2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41085 } } peers { permanent_uuid: "af81bb4b529c4a58824980974cb7362e" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 37615 } } peers { permanent_uuid: "50444827a7fb4637bee08e5afafa8f84" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44565 } }
I20260928 06:35:43.759876 12662 leader_election.cc:288] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84 [CANDIDATE]: Term 1 election: Requested vote from peers b63caecbb37e43f58339a2f1a616de2c (127.5.22.2:41085), af81bb4b529c4a58824980974cb7362e (127.5.22.3:37615)
I20260928 06:35:43.760237 12439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6951ec8504424188944753df2101db70" candidate_uuid: "50444827a7fb4637bee08e5afafa8f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63caecbb37e43f58339a2f1a616de2c"
I20260928 06:35:43.760330 12439 raft_consensus.cc:3037] T 6951ec8504424188944753df2101db70 P b63caecbb37e43f58339a2f1a616de2c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:43.761376 12439 raft_consensus.cc:2454] T 6951ec8504424188944753df2101db70 P b63caecbb37e43f58339a2f1a616de2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50444827a7fb4637bee08e5afafa8f84 in term 1.
I20260928 06:35:43.761637 12575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6951ec8504424188944753df2101db70" candidate_uuid: "50444827a7fb4637bee08e5afafa8f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af81bb4b529c4a58824980974cb7362e"
I20260928 06:35:43.761689 12237 leader_election.cc:302] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50444827a7fb4637bee08e5afafa8f84, b63caecbb37e43f58339a2f1a616de2c; no voters:
I20260928 06:35:43.761767 12575 raft_consensus.cc:3037] T 6951ec8504424188944753df2101db70 P af81bb4b529c4a58824980974cb7362e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:43.761948 12662 raft_consensus.cc:2781] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:43.762147 12662 raft_consensus.cc:686] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84 [term 1 LEADER]: Becoming Leader. State: Replica: 50444827a7fb4637bee08e5afafa8f84, State: Running, Role: LEADER
I20260928 06:35:43.762732 12662 consensus_queue.cc:237] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63caecbb37e43f58339a2f1a616de2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41085 } } peers { permanent_uuid: "af81bb4b529c4a58824980974cb7362e" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 37615 } } peers { permanent_uuid: "50444827a7fb4637bee08e5afafa8f84" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44565 } }
I20260928 06:35:43.763094 12575 raft_consensus.cc:2454] T 6951ec8504424188944753df2101db70 P af81bb4b529c4a58824980974cb7362e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50444827a7fb4637bee08e5afafa8f84 in term 1.
I20260928 06:35:43.766374 11376 catalog_manager.cc:5495] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50444827a7fb4637bee08e5afafa8f84 (127.5.22.1). New cstate: current_term: 1 leader_uuid: "50444827a7fb4637bee08e5afafa8f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63caecbb37e43f58339a2f1a616de2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af81bb4b529c4a58824980974cb7362e" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 37615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50444827a7fb4637bee08e5afafa8f84" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44565 } health_report { overall_health: HEALTHY } } }
W20260928 06:35:43.779227 12351 tablet.cc:2369] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:35:43.801494 12653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.797549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34520 (local address 127.5.22.1:44565)
0928 06:35:43.797702 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:43.797706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:43.797721 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:35:43.797807 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:43.797809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:43.797910 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:43.798018 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:43.798274 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.798931 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.800047 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.800256 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.800337 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.800699 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:43.801132 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:43.801227 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:43.801340 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 06:35:43.801389 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:35:43.807715 12575 raft_consensus.cc:1261] T 6951ec8504424188944753df2101db70 P af81bb4b529c4a58824980974cb7362e [term 1 FOLLOWER]: Refusing update from remote peer 50444827a7fb4637bee08e5afafa8f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:43.807710 12439 raft_consensus.cc:1261] T 6951ec8504424188944753df2101db70 P b63caecbb37e43f58339a2f1a616de2c [term 1 FOLLOWER]: Refusing update from remote peer 50444827a7fb4637bee08e5afafa8f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:43.808488 12662 consensus_queue.cc:1035] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af81bb4b529c4a58824980974cb7362e" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:43.808688 12665 consensus_queue.cc:1035] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63caecbb37e43f58339a2f1a616de2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:43.817497 12669 mvcc.cc:205] Tried to move back new op lower bound from 7334204800228454400 to 7334204800055201792. Current Snapshot: MvccSnapshot[applied={T|T < 7334204800228454400}]
I20260928 06:35:43.828457 11376 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:43.828610 11376 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:43.834218 11376 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5973a583576742e5a79e1b84b9e53783: Sending DeleteTablet for 3 replicas of tablet 6951ec8504424188944753df2101db70
I20260928 06:35:43.835021 12419 tablet_service.cc:1514] Processing DeleteTablet for tablet 6951ec8504424188944753df2101db70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:43 UTC) from {username='oryx', principal='oryx/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:45058
I20260928 06:35:43.835110 12282 tablet_service.cc:1514] Processing DeleteTablet for tablet 6951ec8504424188944753df2101db70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:43 UTC) from {username='oryx', principal='oryx/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:34516
I20260928 06:35:43.835556 12677 tablet_replica.cc:330] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84: stopping tablet replica
I20260928 06:35:43.835918 12677 raft_consensus.cc:2229] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:35:43.836104 12555 tablet_service.cc:1514] Processing DeleteTablet for tablet 6951ec8504424188944753df2101db70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:43 UTC) from {username='oryx', principal='oryx/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:43886
I20260928 06:35:43.836374 12677 raft_consensus.cc:2258] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:43.836566 12679 tablet_replica.cc:330] T 6951ec8504424188944753df2101db70 P af81bb4b529c4a58824980974cb7362e: stopping tablet replica
I20260928 06:35:43.836730 12678 tablet_replica.cc:330] T 6951ec8504424188944753df2101db70 P b63caecbb37e43f58339a2f1a616de2c: stopping tablet replica
I20260928 06:35:43.836992 12679 raft_consensus.cc:2229] T 6951ec8504424188944753df2101db70 P af81bb4b529c4a58824980974cb7362e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:43.836992 12678 raft_consensus.cc:2229] T 6951ec8504424188944753df2101db70 P b63caecbb37e43f58339a2f1a616de2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:43.837299 12678 raft_consensus.cc:2258] T 6951ec8504424188944753df2101db70 P b63caecbb37e43f58339a2f1a616de2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:43.837299 12679 raft_consensus.cc:2258] T 6951ec8504424188944753df2101db70 P af81bb4b529c4a58824980974cb7362e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:43.838969 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 11347
I20260928 06:35:43.839715 12679 ts_tablet_manager.cc:1905] T 6951ec8504424188944753df2101db70 P af81bb4b529c4a58824980974cb7362e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:43.839068 12678 ts_tablet_manager.cc:1905] T 6951ec8504424188944753df2101db70 P b63caecbb37e43f58339a2f1a616de2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:43.842655 12678 ts_tablet_manager.cc:1918] T 6951ec8504424188944753df2101db70 P b63caecbb37e43f58339a2f1a616de2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:43.842655 12679 ts_tablet_manager.cc:1918] T 6951ec8504424188944753df2101db70 P af81bb4b529c4a58824980974cb7362e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:43.842769 12678 log.cc:1196] T 6951ec8504424188944753df2101db70 P b63caecbb37e43f58339a2f1a616de2c: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-1/wal/wals/6951ec8504424188944753df2101db70
I20260928 06:35:43.842769 12679 log.cc:1196] T 6951ec8504424188944753df2101db70 P af81bb4b529c4a58824980974cb7362e: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/6951ec8504424188944753df2101db70
I20260928 06:35:43.843108 12678 ts_tablet_manager.cc:1939] T 6951ec8504424188944753df2101db70 P b63caecbb37e43f58339a2f1a616de2c: Deleting consensus metadata
I20260928 06:35:43.843108 12679 ts_tablet_manager.cc:1939] T 6951ec8504424188944753df2101db70 P af81bb4b529c4a58824980974cb7362e: Deleting consensus metadata
I20260928 06:35:43.846805 12677 ts_tablet_manager.cc:1905] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:35:43.848760 12374 connection.cc:535] server connection from 127.0.0.1:45058 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 06:35:43.849284 12509 connection.cc:535] server connection from 127.0.0.1:43886 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:35:43.849951 12677 ts_tablet_manager.cc:1918] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:43.850056 12677 log.cc:1196] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-0/wal/wals/6951ec8504424188944753df2101db70
I20260928 06:35:43.850454 12677 ts_tablet_manager.cc:1939] T 6951ec8504424188944753df2101db70 P 50444827a7fb4637bee08e5afafa8f84: Deleting consensus metadata
W20260928 06:35:43.851722 12238 connection.cc:441] server connection from 127.0.0.1:34516 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34516 (request call id 1) could send its response
I20260928 06:35:43.855258 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.838304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36496 (local address 127.5.22.61:39427)
0928 06:35:43.838460 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:43.838465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:43.844147 (+ 5682us) server_negotiation.cc:407] Connection header received
0928 06:35:43.844375 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:43.844379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:43.844434 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:43.844494 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:43.845435 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.846180 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.847436 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.847753 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.847832 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.850681 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:43.850686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:43.850688 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:43.850717 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:43.852501 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.853014 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.853018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.853023 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.853156 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.853615 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.853619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.853620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.853883 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:43.854020 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:43.854963 (+ 943us) server_negotiation.cc:299] Negotiation successful
0928 06:35:43.855085 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 06:35:43.864883 12632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.853360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.5.22.61:39427)
0928 06:35:43.853489 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:43.853494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:43.854228 (+ 734us) server_negotiation.cc:407] Connection header received
0928 06:35:43.854435 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:43.854439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:43.854484 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:43.854537 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:43.855648 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.856537 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.858466 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.858888 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.858967 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.860812 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:43.860816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:43.860820 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:43.860846 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:43.862895 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.863418 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.863423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.863425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.863514 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.863841 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.863845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.863847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.864075 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:43.864187 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:43.864562 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 06:35:43.864693 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:35:43.866164 12355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.837354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39380 (local address 127.5.22.60:46045)
0928 06:35:43.837485 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:43.837490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:43.854756 (+ 17266us) server_negotiation.cc:407] Connection header received
0928 06:35:43.854866 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:43.854871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:43.854924 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:43.855007 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:43.856023 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.856921 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.858426 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.858763 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.858855 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.861172 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:43.861177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:43.861180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:43.861207 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:43.863763 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.864420 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.864424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.864426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.864549 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.864876 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.864880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.864882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.865346 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:43.865438 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:43.865746 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 06:35:43.865992 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51,"spinlock_wait_cycles":6528}
I20260928 06:35:43.866199 12688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.854286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39386 (local address 127.5.22.60:46045)
0928 06:35:43.856134 (+ 1848us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:43.856138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:43.856150 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:35:43.856179 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:43.856181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:43.856218 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:43.856297 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:43.857169 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.857972 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.859198 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.859567 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.859681 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.861384 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:43.861389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:43.861394 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:43.861426 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:43.863793 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.864420 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.864424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.864426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.864557 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.864876 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.864880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.864882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.865256 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:43.865378 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:43.865960 (+ 582us) server_negotiation.cc:299] Negotiation successful
0928 06:35:43.866060 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":1775,"spinlock_wait_cycles":6656,"thread_start_us":99,"threads_started":1}
I20260928 06:35:44.822970 12689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:44.822452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.5.22.2:47801)
0928 06:35:44.822750 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:44.822853 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":83,"threads_started":1}
W20260928 06:35:44.823364 12486 heartbeater.cc:644] Failed to heartbeat to 127.5.22.62:38463 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
I20260928 06:35:44.825183 12690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:44.824761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.5.22.1:37321)
0928 06:35:44.824973 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:44.825062 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":65,"threads_started":1}
W20260928 06:35:44.825502 12349 heartbeater.cc:644] Failed to heartbeat to 127.5.22.62:38463 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
I20260928 06:35:44.825887 12691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:44.825499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.5.22.3:50837)
0928 06:35:44.825743 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:44.825814 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":69,"threads_started":1}
W20260928 06:35:44.826154 12622 heartbeater.cc:644] Failed to heartbeat to 127.5.22.62:38463 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
I20260928 06:35:45.333773 12692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5973a583576742e5a79e1b84b9e53783)
I20260928 06:35:45.333976 12692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: false } }
I20260928 06:35:45.334707 12693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5973a583576742e5a79e1b84b9e53783)
I20260928 06:35:45.334910 12693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: false } }
I20260928 06:35:45.335237 12692 leader_election.cc:288] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5973a583576742e5a79e1b84b9e53783 (127.5.22.62:38463), 2181a9b0ee3a437693d113b48de29cef (127.5.22.61:39427)
I20260928 06:35:45.335917 12693 leader_election.cc:288] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5973a583576742e5a79e1b84b9e53783 (127.5.22.62:38463), 032daa7770c44a8aa5d0171610287234 (127.5.22.60:46045)
I20260928 06:35:45.336423 12697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.335974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.0.0.1:48728)
0928 06:35:45.336242 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.336316 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":116,"threads_started":1}
W20260928 06:35:45.336571 11966 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
I20260928 06:35:45.336997 12695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.335413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.0.0.1:48726)
0928 06:35:45.335681 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.335762 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":58,"threads_started":1}
W20260928 06:35:45.337163 12156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
I20260928 06:35:45.337529 12698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.337282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.0.0.1:48742)
0928 06:35:45.337408 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.337465 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 06:35:45.337998 12695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.337743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.0.0.1:48752)
0928 06:35:45.337874 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.337939 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260928 06:35:45.338022 11966 leader_election.cc:334] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5973a583576742e5a79e1b84b9e53783 (127.5.22.62:38463): Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
W20260928 06:35:45.338111 12156 leader_election.cc:334] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5973a583576742e5a79e1b84b9e53783 (127.5.22.62:38463): Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
I20260928 06:35:45.339772 12696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.335504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36512 (local address 127.5.22.61:39427)
0928 06:35:45.335750 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:45.335756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:45.335818 (+ 62us) server_negotiation.cc:407] Connection header received
0928 06:35:45.335996 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:45.336002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:45.336051 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:45.336140 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:45.336456 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.337229 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.338749 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.339262 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.339346 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:45.339392 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:45.339623 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 06:35:45.339681 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":168,"thread_start_us":48,"threads_started":1}
I20260928 06:35:45.339790 12694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.335453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:39427 (local address 127.0.0.1:36512)
0928 06:35:45.335743 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.335759 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:45.335878 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:45.336162 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:45.336165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:45.336182 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:45.336290 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:45.336315 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.337353 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.337360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:45.338625 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:45.338635 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.339381 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.339385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:45.339475 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:45.339502 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 06:35:45.339591 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:35:45.339662 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":71,"threads_started":1}
I20260928 06:35:45.340277 11989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "032daa7770c44a8aa5d0171610287234" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2181a9b0ee3a437693d113b48de29cef" is_pre_election: true
I20260928 06:35:45.340492 11989 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 032daa7770c44a8aa5d0171610287234 in term 1.
I20260928 06:35:45.340889 12153 leader_election.cc:302] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 032daa7770c44a8aa5d0171610287234, 2181a9b0ee3a437693d113b48de29cef; no voters: 5973a583576742e5a79e1b84b9e53783
I20260928 06:35:45.341138 12692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 06:35:45.341215 12692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5973a583576742e5a79e1b84b9e53783)
I20260928 06:35:45.341269 12697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.336343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:46045 (local address 127.0.0.1:39400)
0928 06:35:45.336582 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.336599 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:45.336683 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:45.337737 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:45.337740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:45.337756 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:45.337843 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:45.337848 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.338840 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.338842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:45.340094 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:45.340103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.340856 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.340859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:45.340950 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:45.340970 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 06:35:45.341047 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 06:35:45.341120 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":20,"thread_start_us":58,"threads_started":1}
I20260928 06:35:45.341265 12692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:35:45.341369 12700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.336845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39400 (local address 127.5.22.60:46045)
0928 06:35:45.337021 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:45.337026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:45.337051 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:35:45.337548 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:45.337555 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:45.337608 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:45.337945 (+ 337us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:45.338059 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.338731 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.340232 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.340757 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.340862 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:45.340923 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:45.341063 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 06:35:45.341133 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":48,"threads_started":1}
I20260928 06:35:45.341739 12179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2181a9b0ee3a437693d113b48de29cef" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "032daa7770c44a8aa5d0171610287234" is_pre_election: true
I20260928 06:35:45.342525 12692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: false } }
I20260928 06:35:45.342789 12179 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2181a9b0ee3a437693d113b48de29cef in current term 2: Already voted for candidate 032daa7770c44a8aa5d0171610287234 in this term.
I20260928 06:35:45.342926 12692 leader_election.cc:288] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [CANDIDATE]: Term 2 election: Requested vote from peers 5973a583576742e5a79e1b84b9e53783 (127.5.22.62:38463), 2181a9b0ee3a437693d113b48de29cef (127.5.22.61:39427)
I20260928 06:35:45.343262 11966 leader_election.cc:302] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2181a9b0ee3a437693d113b48de29cef; no voters: 032daa7770c44a8aa5d0171610287234, 5973a583576742e5a79e1b84b9e53783
I20260928 06:35:45.343328 12694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.343049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.0.0.1:48760)
0928 06:35:45.343186 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.343262 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 06:35:45.343277 11989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "032daa7770c44a8aa5d0171610287234" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2181a9b0ee3a437693d113b48de29cef"
I20260928 06:35:45.343425 11989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:35:45.343920 12695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.343710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.0.0.1:48772)
0928 06:35:45.343804 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.343856 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 06:35:45.344022 12156 leader_election.cc:334] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5973a583576742e5a79e1b84b9e53783 (127.5.22.62:38463): Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
I20260928 06:35:45.344554 11989 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 032daa7770c44a8aa5d0171610287234 in term 2.
I20260928 06:35:45.344717 12693 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:35:45.344825 12153 leader_election.cc:302] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 032daa7770c44a8aa5d0171610287234, 2181a9b0ee3a437693d113b48de29cef; no voters: 5973a583576742e5a79e1b84b9e53783
I20260928 06:35:45.345001 12692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:35:45.345229 12692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [term 2 LEADER]: Becoming Leader. State: Replica: 032daa7770c44a8aa5d0171610287234, State: Running, Role: LEADER
I20260928 06:35:45.345487 12692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: false } }
I20260928 06:35:45.346529 12702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 032daa7770c44a8aa5d0171610287234. Latest consensus state: current_term: 2 leader_uuid: "032daa7770c44a8aa5d0171610287234" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: false } } }
I20260928 06:35:45.346688 12702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:45.346967 12704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:45.349037 12704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:45.349445 12704 catalog_manager.cc:1259] Loaded cluster ID: 6c9264db8f9d43f884348bed61629dcc
I20260928 06:35:45.349516 12704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:45.350142 12704 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:45.350656 12704 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234: Loaded TSK: 0
I20260928 06:35:45.351341 12704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:45.387038 12169 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:35:45.390069 12169 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:45.392966 12695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.392742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.0.0.1:48786)
0928 06:35:45.392854 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.392904 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 06:35:45.393236 11989 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [term 2 FOLLOWER]: Refusing update from remote peer 032daa7770c44a8aa5d0171610287234: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 06:35:45.393757 12695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.393544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.0.0.1:48792)
0928 06:35:45.393655 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.393706 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 06:35:45.393778 12702 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260928 06:35:45.393935 12156 consensus_peers.cc:487] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 -> Peer 5973a583576742e5a79e1b84b9e53783 (127.5.22.62:38463): Couldn't send request to peer 5973a583576742e5a79e1b84b9e53783. Status: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 06:35:45.395208 12693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 032daa7770c44a8aa5d0171610287234. Latest consensus state: current_term: 2 leader_uuid: "032daa7770c44a8aa5d0171610287234" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: false } } }
I20260928 06:35:45.395328 12693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:35:45.396392 12702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "032daa7770c44a8aa5d0171610287234" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: false } } }
I20260928 06:35:45.396504 12702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:45.396651 12693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "032daa7770c44a8aa5d0171610287234" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5973a583576742e5a79e1b84b9e53783" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38463 } } peers { permanent_uuid: "2181a9b0ee3a437693d113b48de29cef" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "032daa7770c44a8aa5d0171610287234" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46045 } attrs { promote: false } } }
I20260928 06:35:45.396804 12693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2181a9b0ee3a437693d113b48de29cef [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:35:45.396881 12706 mvcc.cc:205] Tried to move back new op lower bound from 7334204806725017600 to 7334204806537269248. Current Snapshot: MvccSnapshot[applied={T|T < 7334204806725017600}]
I20260928 06:35:45.407471 12711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.402871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34528 (local address 127.5.22.1:44565)
0928 06:35:45.403906 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:45.403911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:45.403925 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:35:45.403981 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:45.403983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:45.404073 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:45.404413 (+ 340us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:45.404519 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.404995 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.406290 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.406916 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.407034 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:45.407101 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:45.407279 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 06:35:45.407355 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":955,"thread_start_us":82,"threads_started":1}
I20260928 06:35:45.407465 12695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.402732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:44565 (local address 127.0.0.1:34528)
0928 06:35:45.402862 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.402879 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:45.402970 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:45.404193 (+ 1223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:45.404195 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:45.404207 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:45.404306 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:45.404314 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.405105 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.405112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:45.406151 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:45.406161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.407033 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.407044 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:45.407144 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:45.407173 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 06:35:45.407261 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:35:45.407332 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":3}
I20260928 06:35:45.408098 12694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.403007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:37615 (local address 127.0.0.1:43898)
0928 06:35:45.403161 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.403179 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:45.403262 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:45.404115 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:45.404121 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:45.404142 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:45.404254 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:45.404262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.405452 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.405460 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:45.406625 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:45.406636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.407647 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.407651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:45.407762 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:45.407816 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 06:35:45.407882 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:35:45.407961 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":25}
I20260928 06:35:45.408269 12713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.403585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:41085 (local address 127.0.0.1:45070)
0928 06:35:45.403808 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.403821 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:45.403905 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:45.404295 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:45.404297 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:45.404338 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:45.404448 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:45.404456 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.405533 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.405536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:45.406677 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:45.406686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.407768 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.407771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:45.407904 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:45.407925 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 06:35:45.407992 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:35:45.408055 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":6,"thread_start_us":98,"threads_started":1}
I20260928 06:35:45.408293 12282 tablet_service.cc:1467] Processing CreateTablet for tablet 6a0fe5e6f25449909ddf8de85637d956 (DEFAULT_TABLE table=test-table [id=9f2d1446fb3e4bedbe6ba3f6509e89a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:45.408401 12714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.403690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45070 (local address 127.5.22.2:41085)
0928 06:35:45.403954 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:45.403959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:45.403972 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:35:45.404027 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:45.404033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:45.404092 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:45.404173 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:45.404587 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.405418 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.406804 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.407491 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.407623 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:45.407686 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:45.408161 (+ 475us) server_negotiation.cc:299] Negotiation successful
0928 06:35:45.408258 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":104,"threads_started":1}
I20260928 06:35:45.408598 12282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a0fe5e6f25449909ddf8de85637d956. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:45.408730 12712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.403288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43898 (local address 127.5.22.3:37615)
0928 06:35:45.403559 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:45.403565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:45.403585 (+ 20us) server_negotiation.cc:407] Connection header received
0928 06:35:45.403931 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:45.403935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:45.403994 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:45.404094 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:45.404371 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.405298 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.406804 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.407499 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.408366 (+ 867us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:45.408461 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:45.408504 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 06:35:45.408583 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":73,"threads_started":1}
I20260928 06:35:45.408943 12419 tablet_service.cc:1467] Processing CreateTablet for tablet 6a0fe5e6f25449909ddf8de85637d956 (DEFAULT_TABLE table=test-table [id=9f2d1446fb3e4bedbe6ba3f6509e89a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:45.409184 12419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a0fe5e6f25449909ddf8de85637d956. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:45.411085 12715 tablet_bootstrap.cc:492] T 6a0fe5e6f25449909ddf8de85637d956 P 50444827a7fb4637bee08e5afafa8f84: Bootstrap starting.
I20260928 06:35:45.411231 12555 tablet_service.cc:1467] Processing CreateTablet for tablet 6a0fe5e6f25449909ddf8de85637d956 (DEFAULT_TABLE table=test-table [id=9f2d1446fb3e4bedbe6ba3f6509e89a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:45.411491 12555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a0fe5e6f25449909ddf8de85637d956. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:45.411646 12716 tablet_bootstrap.cc:492] T 6a0fe5e6f25449909ddf8de85637d956 P b63caecbb37e43f58339a2f1a616de2c: Bootstrap starting.
I20260928 06:35:45.412096 12715 tablet_bootstrap.cc:654] T 6a0fe5e6f25449909ddf8de85637d956 P 50444827a7fb4637bee08e5afafa8f84: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:45.412497 12716 tablet_bootstrap.cc:654] T 6a0fe5e6f25449909ddf8de85637d956 P b63caecbb37e43f58339a2f1a616de2c: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:45.413542 12715 tablet_bootstrap.cc:492] T 6a0fe5e6f25449909ddf8de85637d956 P 50444827a7fb4637bee08e5afafa8f84: No bootstrap required, opened a new log
I20260928 06:35:45.413632 12715 ts_tablet_manager.cc:1397] T 6a0fe5e6f25449909ddf8de85637d956 P 50444827a7fb4637bee08e5afafa8f84: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:35:45.413836 12716 tablet_bootstrap.cc:492] T 6a0fe5e6f25449909ddf8de85637d956 P b63caecbb37e43f58339a2f1a616de2c: No bootstrap required, opened a new log
I20260928 06:35:45.413918 12716 ts_tablet_manager.cc:1397] T 6a0fe5e6f25449909ddf8de85637d956 P b63caecbb37e43f58339a2f1a616de2c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:45.414299 12715 raft_consensus.cc:348] T 6a0fe5e6f25449909ddf8de85637d956 P 50444827a7fb4637bee08e5afafa8f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50444827a7fb4637bee08e5afafa8f84" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44565 } } peers { permanent_uuid: "af81bb4b529c4a58824980974cb7362e" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 37615 } } peers { permanent_uuid: "b63caecbb37e43f58339a2f1a616de2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41085 } }
I20260928 06:35:45.414463 12715 raft_consensus.cc:374] T 6a0fe5e6f25449909ddf8de85637d956 P 50444827a7fb4637bee08e5afafa8f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:45.414505 12715 raft_consensus.cc:729] T 6a0fe5e6f25449909ddf8de85637d956 P 50444827a7fb4637bee08e5afafa8f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50444827a7fb4637bee08e5afafa8f84, State: Initialized, Role: FOLLOWER
I20260928 06:35:45.414506 12719 tablet_bootstrap.cc:492] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e: Bootstrap starting.
I20260928 06:35:45.414474 12716 raft_consensus.cc:348] T 6a0fe5e6f25449909ddf8de85637d956 P b63caecbb37e43f58339a2f1a616de2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50444827a7fb4637bee08e5afafa8f84" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44565 } } peers { permanent_uuid: "af81bb4b529c4a58824980974cb7362e" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 37615 } } peers { permanent_uuid: "b63caecbb37e43f58339a2f1a616de2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41085 } }
I20260928 06:35:45.414575 12716 raft_consensus.cc:374] T 6a0fe5e6f25449909ddf8de85637d956 P b63caecbb37e43f58339a2f1a616de2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:45.414676 12716 raft_consensus.cc:729] T 6a0fe5e6f25449909ddf8de85637d956 P b63caecbb37e43f58339a2f1a616de2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63caecbb37e43f58339a2f1a616de2c, State: Initialized, Role: FOLLOWER
I20260928 06:35:45.414671 12715 consensus_queue.cc:260] T 6a0fe5e6f25449909ddf8de85637d956 P 50444827a7fb4637bee08e5afafa8f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50444827a7fb4637bee08e5afafa8f84" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44565 } } peers { permanent_uuid: "af81bb4b529c4a58824980974cb7362e" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 37615 } } peers { permanent_uuid: "b63caecbb37e43f58339a2f1a616de2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41085 } }
I20260928 06:35:45.414808 12716 consensus_queue.cc:260] T 6a0fe5e6f25449909ddf8de85637d956 P b63caecbb37e43f58339a2f1a616de2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50444827a7fb4637bee08e5afafa8f84" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44565 } } peers { permanent_uuid: "af81bb4b529c4a58824980974cb7362e" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 37615 } } peers { permanent_uuid: "b63caecbb37e43f58339a2f1a616de2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41085 } }
I20260928 06:35:45.415079 12715 ts_tablet_manager.cc:1428] T 6a0fe5e6f25449909ddf8de85637d956 P 50444827a7fb4637bee08e5afafa8f84: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:45.415199 12716 ts_tablet_manager.cc:1428] T 6a0fe5e6f25449909ddf8de85637d956 P b63caecbb37e43f58339a2f1a616de2c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:45.415692 12719 tablet_bootstrap.cc:654] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:45.416738 12722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.415529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.5.22.1:43905)
0928 06:35:45.416569 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.416631 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":972,"thread_start_us":100,"threads_started":1}
I20260928 06:35:45.418540 12722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.418260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.5.22.1:57969)
0928 06:35:45.418406 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.418468 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 06:35:45.418742 12348 heartbeater.cc:503] Master 127.5.22.60:46045 was elected leader, sending a full tablet report...
I20260928 06:35:45.418826 12485 heartbeater.cc:503] Master 127.5.22.60:46045 was elected leader, sending a full tablet report...
I20260928 06:35:45.419203 12723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.415668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.5.22.2:40519)
0928 06:35:45.419024 (+ 3356us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.419091 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3280,"thread_start_us":97,"threads_started":1}
I20260928 06:35:45.420238 12723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.419646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.5.22.2:37359)
0928 06:35:45.420132 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.420185 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416}
I20260928 06:35:45.420858 12719 tablet_bootstrap.cc:492] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e: No bootstrap required, opened a new log
I20260928 06:35:45.420972 12719 ts_tablet_manager.cc:1397] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260928 06:35:45.421406 12719 raft_consensus.cc:348] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50444827a7fb4637bee08e5afafa8f84" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44565 } } peers { permanent_uuid: "af81bb4b529c4a58824980974cb7362e" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 37615 } } peers { permanent_uuid: "b63caecbb37e43f58339a2f1a616de2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41085 } }
I20260928 06:35:45.421540 12719 raft_consensus.cc:374] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:45.421589 12719 raft_consensus.cc:729] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af81bb4b529c4a58824980974cb7362e, State: Initialized, Role: FOLLOWER
I20260928 06:35:45.421730 12719 consensus_queue.cc:260] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50444827a7fb4637bee08e5afafa8f84" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44565 } } peers { permanent_uuid: "af81bb4b529c4a58824980974cb7362e" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 37615 } } peers { permanent_uuid: "b63caecbb37e43f58339a2f1a616de2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41085 } }
I20260928 06:35:45.422091 12719 ts_tablet_manager.cc:1428] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:45.422931 12726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.422528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.5.22.3:34959)
0928 06:35:45.422770 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.422818 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":97,"threads_started":1}
I20260928 06:35:45.423790 12621 heartbeater.cc:503] Master 127.5.22.60:46045 was elected leader, sending a full tablet report...
I20260928 06:35:45.424226 12726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.423337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.5.22.3:57673)
0928 06:35:45.424097 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.424157 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219}
I20260928 06:35:45.640682 12725 raft_consensus.cc:482] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:45.640849 12725 raft_consensus.cc:504] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50444827a7fb4637bee08e5afafa8f84" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44565 } } peers { permanent_uuid: "af81bb4b529c4a58824980974cb7362e" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 37615 } } peers { permanent_uuid: "b63caecbb37e43f58339a2f1a616de2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41085 } }
I20260928 06:35:45.641911 12725 leader_election.cc:288] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50444827a7fb4637bee08e5afafa8f84 (127.5.22.1:44565), b63caecbb37e43f58339a2f1a616de2c (127.5.22.2:41085)
I20260928 06:35:45.646504 12726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.642203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:44565 (local address 127.5.22.3:48775)
0928 06:35:45.642377 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.642394 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:45.642509 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:45.642854 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:45.642857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:45.642879 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:45.643038 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:45.643047 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.643959 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.643962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:45.645270 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:45.645279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.646077 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.646080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:45.646163 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:45.646184 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 06:35:45.646245 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:35:45.646356 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":46}
I20260928 06:35:45.646525 12711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.642290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:48775 (local address 127.5.22.1:44565)
0928 06:35:45.642439 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:45.642446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:45.642462 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:35:45.642642 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:45.642645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:45.642700 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:45.642789 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:45.643150 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.643845 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.645392 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.645947 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.646053 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:45.646127 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:45.646293 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 06:35:45.646389 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 06:35:45.646776 12714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.642691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:40163 (local address 127.5.22.2:41085)
0928 06:35:45.642846 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:45.642850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:45.642974 (+ 124us) server_negotiation.cc:407] Connection header received
0928 06:35:45.643076 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:45.643078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:45.643148 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:45.643232 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:45.643638 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.644243 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.645592 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.646194 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.646332 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:45.646389 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:45.646580 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 06:35:45.646659 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 06:35:45.646799 12727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.642556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:41085 (local address 127.5.22.3:40163)
0928 06:35:45.642881 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.642895 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:45.642986 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:45.643310 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:45.643313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:45.643330 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:45.643505 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:45.643512 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.644362 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.644365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:45.645479 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:45.645485 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.646353 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.646357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:45.646440 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:45.646461 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 06:35:45.646564 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 06:35:45.646637 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":25,"thread_start_us":132,"threads_started":1}
I20260928 06:35:45.647022 12302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a0fe5e6f25449909ddf8de85637d956" candidate_uuid: "af81bb4b529c4a58824980974cb7362e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50444827a7fb4637bee08e5afafa8f84" is_pre_election: true
I20260928 06:35:45.647294 12302 raft_consensus.cc:2454] T 6a0fe5e6f25449909ddf8de85637d956 P 50444827a7fb4637bee08e5afafa8f84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af81bb4b529c4a58824980974cb7362e in term 0.
I20260928 06:35:45.647358 12439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a0fe5e6f25449909ddf8de85637d956" candidate_uuid: "af81bb4b529c4a58824980974cb7362e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63caecbb37e43f58339a2f1a616de2c" is_pre_election: true
I20260928 06:35:45.647541 12439 raft_consensus.cc:2454] T 6a0fe5e6f25449909ddf8de85637d956 P b63caecbb37e43f58339a2f1a616de2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af81bb4b529c4a58824980974cb7362e in term 0.
I20260928 06:35:45.647733 12510 leader_election.cc:302] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50444827a7fb4637bee08e5afafa8f84, af81bb4b529c4a58824980974cb7362e; no voters:
I20260928 06:35:45.647964 12725 raft_consensus.cc:2781] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:45.648044 12725 raft_consensus.cc:482] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:45.648100 12725 raft_consensus.cc:3037] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:45.649127 12725 raft_consensus.cc:504] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50444827a7fb4637bee08e5afafa8f84" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44565 } } peers { permanent_uuid: "af81bb4b529c4a58824980974cb7362e" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 37615 } } peers { permanent_uuid: "b63caecbb37e43f58339a2f1a616de2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41085 } }
I20260928 06:35:45.649463 12725 leader_election.cc:288] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e [CANDIDATE]: Term 1 election: Requested vote from peers 50444827a7fb4637bee08e5afafa8f84 (127.5.22.1:44565), b63caecbb37e43f58339a2f1a616de2c (127.5.22.2:41085)
I20260928 06:35:45.649808 12302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a0fe5e6f25449909ddf8de85637d956" candidate_uuid: "af81bb4b529c4a58824980974cb7362e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50444827a7fb4637bee08e5afafa8f84"
I20260928 06:35:45.649893 12439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a0fe5e6f25449909ddf8de85637d956" candidate_uuid: "af81bb4b529c4a58824980974cb7362e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63caecbb37e43f58339a2f1a616de2c"
I20260928 06:35:45.649948 12302 raft_consensus.cc:3037] T 6a0fe5e6f25449909ddf8de85637d956 P 50444827a7fb4637bee08e5afafa8f84 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:45.650029 12439 raft_consensus.cc:3037] T 6a0fe5e6f25449909ddf8de85637d956 P b63caecbb37e43f58339a2f1a616de2c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:45.651063 12439 raft_consensus.cc:2454] T 6a0fe5e6f25449909ddf8de85637d956 P b63caecbb37e43f58339a2f1a616de2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af81bb4b529c4a58824980974cb7362e in term 1.
I20260928 06:35:45.651069 12302 raft_consensus.cc:2454] T 6a0fe5e6f25449909ddf8de85637d956 P 50444827a7fb4637bee08e5afafa8f84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af81bb4b529c4a58824980974cb7362e in term 1.
I20260928 06:35:45.651342 12510 leader_election.cc:302] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50444827a7fb4637bee08e5afafa8f84, af81bb4b529c4a58824980974cb7362e; no voters:
I20260928 06:35:45.651628 12725 raft_consensus.cc:2781] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:45.651893 12725 raft_consensus.cc:686] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e [term 1 LEADER]: Becoming Leader. State: Replica: af81bb4b529c4a58824980974cb7362e, State: Running, Role: LEADER
I20260928 06:35:45.652383 12725 consensus_queue.cc:237] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50444827a7fb4637bee08e5afafa8f84" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44565 } } peers { permanent_uuid: "af81bb4b529c4a58824980974cb7362e" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 37615 } } peers { permanent_uuid: "b63caecbb37e43f58339a2f1a616de2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41085 } }
I20260928 06:35:45.652643 12727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.652386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.5.22.3:60223)
0928 06:35:45.652494 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.652574 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 06:35:45.654882 12168 catalog_manager.cc:5495] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e reported cstate change: term changed from 0 to 1, leader changed from <none> to af81bb4b529c4a58824980974cb7362e (127.5.22.3). New cstate: current_term: 1 leader_uuid: "af81bb4b529c4a58824980974cb7362e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50444827a7fb4637bee08e5afafa8f84" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af81bb4b529c4a58824980974cb7362e" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 37615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b63caecbb37e43f58339a2f1a616de2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41085 } health_report { overall_health: UNKNOWN } } }
I20260928 06:35:45.699309 12712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.695692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43900 (local address 127.5.22.3:37615)
0928 06:35:45.695832 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:45.695838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:45.695852 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:35:45.695960 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:45.695963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:45.696018 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:45.696130 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:45.696361 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.696873 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.697717 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:45.697997 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:45.698124 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:45.698468 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:45.698893 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:45.699018 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:45.699112 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 06:35:45.699166 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 06:35:45.701872 12439 raft_consensus.cc:1261] T 6a0fe5e6f25449909ddf8de85637d956 P b63caecbb37e43f58339a2f1a616de2c [term 1 FOLLOWER]: Refusing update from remote peer af81bb4b529c4a58824980974cb7362e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:45.702265 12302 raft_consensus.cc:1261] T 6a0fe5e6f25449909ddf8de85637d956 P 50444827a7fb4637bee08e5afafa8f84 [term 1 FOLLOWER]: Refusing update from remote peer af81bb4b529c4a58824980974cb7362e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:45.702399 12725 consensus_queue.cc:1035] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63caecbb37e43f58339a2f1a616de2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:45.702847 12728 consensus_queue.cc:1035] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e [LEADER]: Connected to new peer: Peer: permanent_uuid: "50444827a7fb4637bee08e5afafa8f84" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:45.703979 12723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.703692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.5.22.2:44905)
0928 06:35:45.703817 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.703885 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 06:35:45.706768 12722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.705985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.5.22.1:55305)
0928 06:35:45.706638 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.706694 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":581}
I20260928 06:35:45.706837 12727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.706436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.5.22.3:55963)
0928 06:35:45.706719 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.706771 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213}
I20260928 06:35:45.708168 12727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.707915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.5.22.3:53687)
0928 06:35:45.708053 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.708111 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 06:35:45.708868 12722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.708606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.5.22.1:56577)
0928 06:35:45.708755 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.708803 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 06:35:45.709406 12734 mvcc.cc:205] Tried to move back new op lower bound from 7334204807990157312 to 7334204807794860032. Current Snapshot: MvccSnapshot[applied={T|T < 7334204807990157312}]
I20260928 06:35:45.709558 12732 mvcc.cc:205] Tried to move back new op lower bound from 7334204807990157312 to 7334204807794860032. Current Snapshot: MvccSnapshot[applied={T|T < 7334204807990157312}]
I20260928 06:35:45.714188 12723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.713938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38463 (local address 127.5.22.2:60127)
0928 06:35:45.714057 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.714114 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 06:35:45.716073 12168 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:45.716220 12168 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:45.720144 12168 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 032daa7770c44a8aa5d0171610287234: Sending DeleteTablet for 3 replicas of tablet 6a0fe5e6f25449909ddf8de85637d956
I20260928 06:35:45.720708 12282 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a0fe5e6f25449909ddf8de85637d956 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:45 UTC) from {username='oryx', principal='oryx/127.5.22.60@KRBTEST.COM'} at 127.0.0.1:34528
I20260928 06:35:45.720791 12419 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a0fe5e6f25449909ddf8de85637d956 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:45 UTC) from {username='oryx', principal='oryx/127.5.22.60@KRBTEST.COM'} at 127.0.0.1:45070
I20260928 06:35:45.721001 12555 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a0fe5e6f25449909ddf8de85637d956 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:45 UTC) from {username='oryx', principal='oryx/127.5.22.60@KRBTEST.COM'} at 127.0.0.1:43898
I20260928 06:35:45.721243 12740 tablet_replica.cc:330] T 6a0fe5e6f25449909ddf8de85637d956 P 50444827a7fb4637bee08e5afafa8f84: stopping tablet replica
I20260928 06:35:45.721376 12740 raft_consensus.cc:2229] T 6a0fe5e6f25449909ddf8de85637d956 P 50444827a7fb4637bee08e5afafa8f84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:45.721403 12742 tablet_replica.cc:330] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e: stopping tablet replica
I20260928 06:35:45.721521 12740 raft_consensus.cc:2258] T 6a0fe5e6f25449909ddf8de85637d956 P 50444827a7fb4637bee08e5afafa8f84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:45.721657 12742 raft_consensus.cc:2229] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:35:45.721977 12742 raft_consensus.cc:2258] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:45.722090 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 12123
I20260928 06:35:45.722368 12740 ts_tablet_manager.cc:1905] T 6a0fe5e6f25449909ddf8de85637d956 P 50444827a7fb4637bee08e5afafa8f84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:45.723349 12742 ts_tablet_manager.cc:1905] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:45.725948 12742 ts_tablet_manager.cc:1918] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:45.726101 12742 log.cc:1196] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/6a0fe5e6f25449909ddf8de85637d956
I20260928 06:35:45.726502 12742 ts_tablet_manager.cc:1939] T 6a0fe5e6f25449909ddf8de85637d956 P af81bb4b529c4a58824980974cb7362e: Deleting consensus metadata
I20260928 06:35:45.727249 12156 catalog_manager.cc:4841] TS af81bb4b529c4a58824980974cb7362e (127.5.22.3:37615): tablet 6a0fe5e6f25449909ddf8de85637d956 (table test-table [id=9f2d1446fb3e4bedbe6ba3f6509e89a5]) successfully deleted
I20260928 06:35:45.729326 12741 tablet_replica.cc:330] T 6a0fe5e6f25449909ddf8de85637d956 P b63caecbb37e43f58339a2f1a616de2c: stopping tablet replica
I20260928 06:35:45.729482 12741 raft_consensus.cc:2229] T 6a0fe5e6f25449909ddf8de85637d956 P b63caecbb37e43f58339a2f1a616de2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:45.729642 12741 raft_consensus.cc:2258] T 6a0fe5e6f25449909ddf8de85637d956 P b63caecbb37e43f58339a2f1a616de2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:45.730371 12741 ts_tablet_manager.cc:1905] T 6a0fe5e6f25449909ddf8de85637d956 P b63caecbb37e43f58339a2f1a616de2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:45.731617 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 12356
I20260928 06:35:45.731881 12713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.731530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:44565 (local address 127.0.0.1:34536)
0928 06:35:45.731726 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.731788 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.1:44565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260928 06:35:45.732472 12155 catalog_manager.cc:4584] TS 50444827a7fb4637bee08e5afafa8f84 (127.5.22.1:44565): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6a0fe5e6f25449909ddf8de85637d956: Network error: Client connection negotiation failed: client connection to 127.5.22.1:44565: connect: Connection refused (error 111)
I20260928 06:35:45.738009 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 12492
I20260928 06:35:45.738067 12713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:45.737779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:41085 (local address 127.0.0.1:45072)
0928 06:35:45.737921 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:45.737977 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.2:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 06:35:45.743801 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 11843
I20260928 06:35:45.749396 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 11951
2026-09-28T06:35:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5690 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:45 dist-test-slave-j0qf krb5kdc[12745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:45 dist-test-slave-j0qf krb5kdc[12745](info): set up 2 sockets
Sep 28 06:35:45 dist-test-slave-j0qf krb5kdc[12745](info): commencing operation
krb5kdc: starting...
W20260928 06:35:46.837807 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:35:46 dist-test-slave-j0qf krb5kdc[12745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.62.keytab.
Entry for principal oryx/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.62.keytab.
I20260928 06:35:46.970173 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:42115
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.62.keytab
--principal=oryx/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:42115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:47.088203 12758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:47.088479 12758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:47.088519 12758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:47.092815 12758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:47.092880 12758 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 06:35:47.092898 12758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:47.092917 12758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:47.092960 12758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:47.092993 12758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:47.098600 12758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:42115
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.62.keytab
--principal=oryx/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:42115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:47.099695 12758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:47.100739 12758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:47.109359 12763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:47.109359 12764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:47.109359 12766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:47.109620 12758 server_base.cc:1034] running on GCE node
I20260928 06:35:47.110190 12758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:47.111524 12758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:47.112767 12758 hybrid_clock.cc:648] HybridClock initialized: now 1790577347112732 us; error 49 us; skew 500 ppm
Sep 28 06:35:47 dist-test-slave-j0qf krb5kdc[12745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577347, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:47.115857 12758 init.cc:377] Logged in from keytab as oryx/127.5.22.62@KRBTEST.COM (short username oryx)
I20260928 06:35:47.116799 12758 webserver.cc:460] Webserver started at http://127.5.22.62:37059/ using document root <none> and password file <none>
I20260928 06:35:47.117389 12758 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:47.117501 12758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:47.117719 12758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:47.119838 12758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "447d544f4e8c4b0a88a28db7251bbbd2"
format_stamp: "Formatted at 2026-09-28 06:35:47 on dist-test-slave-j0qf"
server_key: "bfabad45739558b4f2e181c54aa25e0c"
server_key_iv: "baa6d6e77f3b6a5057a912d7ccd63b38"
server_key_version: "encryptionkey@0"
I20260928 06:35:47.120437 12758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "447d544f4e8c4b0a88a28db7251bbbd2"
format_stamp: "Formatted at 2026-09-28 06:35:47 on dist-test-slave-j0qf"
server_key: "bfabad45739558b4f2e181c54aa25e0c"
server_key_iv: "baa6d6e77f3b6a5057a912d7ccd63b38"
server_key_version: "encryptionkey@0"
I20260928 06:35:47.124375 12758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:35:47.126848 12773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:47.128118 12758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:35:47.128275 12758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "447d544f4e8c4b0a88a28db7251bbbd2"
format_stamp: "Formatted at 2026-09-28 06:35:47 on dist-test-slave-j0qf"
server_key: "bfabad45739558b4f2e181c54aa25e0c"
server_key_iv: "baa6d6e77f3b6a5057a912d7ccd63b38"
server_key_version: "encryptionkey@0"
I20260928 06:35:47.128422 12758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:47.162971 12758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:47.166180 12758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:47.166427 12758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:47.174117 12758 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:42115
I20260928 06:35:47.174191 12835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:42115 every 8 connection(s)
I20260928 06:35:47.175125 12758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:47.175733 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 12758
I20260928 06:35:47.175884 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:35:47.176167 5208 external_mini_cluster.cc:1428] Setting key 9581876f59bf729ed8cbabef60887426
I20260928 06:35:47.178539 12836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:47.183867 12836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2: Bootstrap starting.
Sep 28 06:35:47 dist-test-slave-j0qf krb5kdc[12745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM
I20260928 06:35:47.186147 12836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:47.186923 12836 log.cc:824] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:47.188928 12836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2: No bootstrap required, opened a new log
I20260928 06:35:47.190929 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:47.177563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41302 (local address 127.5.22.62:42115)
0928 06:35:47.178018 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:47.178055 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:47.178093 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:35:47.178849 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:47.178867 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:47.179201 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:47.179533 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:47.180364 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.181354 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.182218 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.182746 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.182913 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:47.185411 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:47.185417 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:47.185428 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:47.185455 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:47.187943 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:47.188408 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:47.188414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:47.188427 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:47.188542 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:47.188835 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:47.188840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:47.188842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:47.189288 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:47.189479 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:47.189786 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 06:35:47.190108 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":134,"threads_started":1}
I20260928 06:35:47.192157 12836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447d544f4e8c4b0a88a28db7251bbbd2" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 42115 } }
I20260928 06:35:47.192443 12836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:47.192514 12836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 447d544f4e8c4b0a88a28db7251bbbd2, State: Initialized, Role: FOLLOWER
I20260928 06:35:47.192958 12836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447d544f4e8c4b0a88a28db7251bbbd2" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 42115 } }
I20260928 06:35:47.193135 12836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:47.193207 12836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:47.193305 12836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:47.194216 12836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447d544f4e8c4b0a88a28db7251bbbd2" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 42115 } }
I20260928 06:35:47.194603 12836 leader_election.cc:302] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 447d544f4e8c4b0a88a28db7251bbbd2; no voters:
I20260928 06:35:47.194998 12836 leader_election.cc:288] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:47.195143 12841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:47.195381 12841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2 [term 1 LEADER]: Becoming Leader. State: Replica: 447d544f4e8c4b0a88a28db7251bbbd2, State: Running, Role: LEADER
I20260928 06:35:47.195686 12841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447d544f4e8c4b0a88a28db7251bbbd2" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 42115 } }
I20260928 06:35:47.196328 12836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:47.197714 12843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 447d544f4e8c4b0a88a28db7251bbbd2. Latest consensus state: current_term: 1 leader_uuid: "447d544f4e8c4b0a88a28db7251bbbd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447d544f4e8c4b0a88a28db7251bbbd2" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 42115 } } }
I20260928 06:35:47.197891 12843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:47.198429 12842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "447d544f4e8c4b0a88a28db7251bbbd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447d544f4e8c4b0a88a28db7251bbbd2" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 42115 } } }
I20260928 06:35:47.198550 12842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2 [sys.catalog]: This master's current role is: LEADER
W20260928 06:35:47.201709 12856 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:35:47.201777 12856 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:35:47.201860 12853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:47.202682 12853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:47.207732 12853 catalog_manager.cc:1347] Generated new cluster ID: 43a764ebaea449f5bf32e259dd959a52
I20260928 06:35:47.207793 12853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:47.228801 12853 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:47.229720 12853 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:47.240044 12853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2: Generated new TSK 0
I20260928 06:35:47.240746 12853 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.22.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.22.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.22.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.1.keytab.
Entry for principal oryx/127.5.22.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.1.keytab.
I20260928 06:35:47.283417 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:42115
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.1.keytab
--principal=oryx/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:47.413503 12862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:47.413769 12862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:47.413812 12862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:47.417788 12862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:47.417850 12862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:47.417889 12862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:35:47.417956 12862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:35:47.423568 12862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.1.keytab
--principal=oryx/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.22.62:42115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:47.424619 12862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:47.425649 12862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:47.435245 12870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:47.435237 12867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:47.435237 12868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:47.435606 12862 server_base.cc:1034] running on GCE node
I20260928 06:35:47.435997 12862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:47.436614 12862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:47.437798 12862 hybrid_clock.cc:648] HybridClock initialized: now 1790577347437788 us; error 33 us; skew 500 ppm
Sep 28 06:35:47 dist-test-slave-j0qf krb5kdc[12745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577347, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:47.440564 12862 init.cc:377] Logged in from keytab as oryx/127.5.22.1@KRBTEST.COM (short username oryx)
I20260928 06:35:47.441339 12862 webserver.cc:460] Webserver started at http://127.5.22.1:35427/ using document root <none> and password file <none>
I20260928 06:35:47.441831 12862 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:47.441887 12862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:47.442050 12862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:47.444023 12862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9"
format_stamp: "Formatted at 2026-09-28 06:35:47 on dist-test-slave-j0qf"
server_key: "00199e9ec5670ae923bb77f8c4902408"
server_key_iv: "ef65c9a13567210480fbec5d4191a8b1"
server_key_version: "encryptionkey@0"
I20260928 06:35:47.444542 12862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9"
format_stamp: "Formatted at 2026-09-28 06:35:47 on dist-test-slave-j0qf"
server_key: "00199e9ec5670ae923bb77f8c4902408"
server_key_iv: "ef65c9a13567210480fbec5d4191a8b1"
server_key_version: "encryptionkey@0"
I20260928 06:35:47.448163 12862 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260928 06:35:47.450579 12877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:47.451795 12862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 06:35:47.451918 12862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9"
format_stamp: "Formatted at 2026-09-28 06:35:47 on dist-test-slave-j0qf"
server_key: "00199e9ec5670ae923bb77f8c4902408"
server_key_iv: "ef65c9a13567210480fbec5d4191a8b1"
server_key_version: "encryptionkey@0"
I20260928 06:35:47.452025 12862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:47.494670 12862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:47.498394 12862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:47.498642 12862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:47.500234 12862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:47.500283 12862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:47.500324 12862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:47.500339 12862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:35:47 dist-test-slave-j0qf krb5kdc[12745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577347, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM
I20260928 06:35:47.511528 12862 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:43981
I20260928 06:35:47.511746 12992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:43981 every 8 connection(s)
I20260928 06:35:47.512734 12862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:47.515439 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:47.501208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:50473 (local address 127.5.22.62:42115)
0928 06:35:47.501379 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:47.501384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:47.502383 (+ 999us) server_negotiation.cc:407] Connection header received
0928 06:35:47.503798 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:47.503805 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:47.503869 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:47.504001 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:47.505659 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.506752 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.508156 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.508447 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.508573 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:47.511450 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:47.511455 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:47.511458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:47.511489 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:47.513091 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:47.513747 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:47.513754 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:47.513758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:47.513849 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:47.514234 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:47.514240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:47.514242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:47.514498 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:47.514615 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:47.515120 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 06:35:47.515251 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 06:35:47.516233 12886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:47.501601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:42115 (local address 127.5.22.1:50473)
0928 06:35:47.502156 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:47.502216 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:47.503534 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:47.504208 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:47.504218 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:47.504705 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:47.505477 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:47.505499 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.506882 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.506887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:47.508019 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:47.508029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.508620 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.508624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:47.508810 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:47.509477 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:47.509519 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:47.511256 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:47.513216 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:47.513223 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:47.513238 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:47.513636 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:47.513972 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:47.513976 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:47.513978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:47.514129 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:47.514612 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:47.514620 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:47.514869 (+ 249us) client_negotiation.cc:769] Sending connection context
0928 06:35:47.515060 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 06:35:47.515381 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":144,"threads_started":1}
I20260928 06:35:47.517736 12993 heartbeater.cc:347] Connected to a master server at 127.5.22.62:42115
I20260928 06:35:47.517899 12993 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:47.518517 12993 heartbeater.cc:511] Master 127.5.22.62:42115 requested a full tablet report, sending...
I20260928 06:35:47.520035 12788 ts_manager.cc:194] Registered new tserver with Master: e1c6df1bbd4548d9ab9d6b96a197f4e9 (127.5.22.1:43981)
I20260928 06:35:47.520989 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 12862
I20260928 06:35:47.521133 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:47.521396 12788 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.22.1@KRBTEST.COM'} at 127.5.22.1:50473
I20260928 06:35:47.521466 5208 external_mini_cluster.cc:1428] Setting key 2a33b4b4ef4d20c309915dd2eeba0e22
I20260928 06:35:47.528163 12999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:47.519908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:42115 (local address 127.5.22.1:46277)
0928 06:35:47.520277 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:47.520293 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:47.520388 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:47.520621 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:47.520624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:47.520829 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:47.520941 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:47.520948 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.522047 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.522053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:47.522825 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:47.522835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.523370 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.523372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:47.523546 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:47.524058 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:47.524073 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:47.524521 (+ 448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:47.526587 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:47.526592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:47.526598 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:47.526906 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:47.527188 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:47.527192 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:47.527194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:47.527304 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:47.527667 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:47.527674 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:47.527764 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 06:35:47.527895 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 06:35:47.528007 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":102,"threads_started":1}
I20260928 06:35:47.528405 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:47.519980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:46277 (local address 127.5.22.62:42115)
0928 06:35:47.520121 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:47.520126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:47.520373 (+ 247us) server_negotiation.cc:407] Connection header received
0928 06:35:47.520472 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:47.520474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:47.520514 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:47.520583 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:47.521083 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.521886 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.522945 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.523250 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.523361 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:47.524676 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:47.524683 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:47.524689 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:47.524717 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:47.526476 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:47.527008 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:47.527010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:47.527012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:47.527093 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:47.527411 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:47.527413 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:47.527414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:47.527567 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:47.527639 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:47.527979 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 06:35:47.528136 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for oryx/127.5.22.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.22.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.22.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.2.keytab.
Entry for principal oryx/127.5.22.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.2.keytab.
I20260928 06:35:47.553642 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:42115
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.2.keytab
--principal=oryx/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:47.663856 13002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:47.664137 13002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:47.664212 13002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:47.668211 13002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:47.668295 13002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:47.668370 13002 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:35:47.668452 13002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:35:47.673571 13002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.2.keytab
--principal=oryx/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.22.62:42115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:47.674654 13002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:47.675663 13002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:47.685324 13010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:47.685333 13008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:47.685333 13007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:47.685499 13002 server_base.cc:1034] running on GCE node
I20260928 06:35:47.685973 13002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:47.686630 13002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:47.687805 13002 hybrid_clock.cc:648] HybridClock initialized: now 1790577347687770 us; error 43 us; skew 500 ppm
Sep 28 06:35:47 dist-test-slave-j0qf krb5kdc[12745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577347, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:47.690472 13002 init.cc:377] Logged in from keytab as oryx/127.5.22.2@KRBTEST.COM (short username oryx)
I20260928 06:35:47.691354 13002 webserver.cc:460] Webserver started at http://127.5.22.2:46231/ using document root <none> and password file <none>
I20260928 06:35:47.691895 13002 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:47.691954 13002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:47.692170 13002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:47.694172 13002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "f6179c451b334f648ae3355cca58bb68"
format_stamp: "Formatted at 2026-09-28 06:35:47 on dist-test-slave-j0qf"
server_key: "fc7614d13663dccdc634bc9bbc888e1d"
server_key_iv: "a9dc50759a396bdf4a2348e7d504c0d3"
server_key_version: "encryptionkey@0"
I20260928 06:35:47.694798 13002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "f6179c451b334f648ae3355cca58bb68"
format_stamp: "Formatted at 2026-09-28 06:35:47 on dist-test-slave-j0qf"
server_key: "fc7614d13663dccdc634bc9bbc888e1d"
server_key_iv: "a9dc50759a396bdf4a2348e7d504c0d3"
server_key_version: "encryptionkey@0"
I20260928 06:35:47.698999 13002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 06:35:47.701398 13017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:47.702816 13002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:35:47.702960 13002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "f6179c451b334f648ae3355cca58bb68"
format_stamp: "Formatted at 2026-09-28 06:35:47 on dist-test-slave-j0qf"
server_key: "fc7614d13663dccdc634bc9bbc888e1d"
server_key_iv: "a9dc50759a396bdf4a2348e7d504c0d3"
server_key_version: "encryptionkey@0"
I20260928 06:35:47.703071 13002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:47.714773 13002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:47.717584 13002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:47.717777 13002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:47.719257 13002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:47.719304 13002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:47.719367 13002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:47.719399 13002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:35:47 dist-test-slave-j0qf krb5kdc[12745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577347, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM
I20260928 06:35:47.730901 13002 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:40931
I20260928 06:35:47.731194 13132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:40931 every 8 connection(s)
I20260928 06:35:47.732060 13002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:47.734163 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:47.720187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:38019 (local address 127.5.22.62:42115)
0928 06:35:47.720314 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:47.720318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:47.722743 (+ 2425us) server_negotiation.cc:407] Connection header received
0928 06:35:47.723822 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:47.723825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:47.723896 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:47.723999 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:47.725180 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.725712 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.727086 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.727313 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.727430 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:47.729972 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:47.729975 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:47.729977 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:47.730001 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:47.731778 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:47.732501 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:47.732504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:47.732506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:47.732650 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:47.733114 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:47.733120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:47.733123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:47.733370 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:47.733461 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:47.733952 (+ 491us) server_negotiation.cc:299] Negotiation successful
0928 06:35:47.734048 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:35:47.734951 13026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:47.720752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:42115 (local address 127.5.22.2:38019)
0928 06:35:47.722590 (+ 1838us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:47.722641 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:47.723619 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:47.724172 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:47.724178 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:47.724505 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:47.725035 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:47.725048 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.725836 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.725840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:47.726955 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:47.726964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.727480 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.727484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:47.727646 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:47.728274 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:47.728291 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:47.729796 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:47.731922 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:47.731927 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:47.731937 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:47.732329 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:47.732831 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:47.732836 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:47.732838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:47.733010 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:47.733473 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:47.733478 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:47.733747 (+ 269us) client_negotiation.cc:769] Sending connection context
0928 06:35:47.733896 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 06:35:47.734126 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":143,"threads_started":1}
I20260928 06:35:47.736227 13133 heartbeater.cc:347] Connected to a master server at 127.5.22.62:42115
I20260928 06:35:47.736358 13133 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:47.736902 13133 heartbeater.cc:511] Master 127.5.22.62:42115 requested a full tablet report, sending...
I20260928 06:35:47.737982 12787 ts_manager.cc:194] Registered new tserver with Master: f6179c451b334f648ae3355cca58bb68 (127.5.22.2:40931)
I20260928 06:35:47.738858 12787 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.22.2@KRBTEST.COM'} at 127.5.22.2:38019
I20260928 06:35:47.740036 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 13002
I20260928 06:35:47.740128 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:47.740430 5208 external_mini_cluster.cc:1428] Setting key d65c3efb1c49f6e7ec1e96b196a2a437
I20260928 06:35:47.746765 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:47.738779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:46881 (local address 127.5.22.62:42115)
0928 06:35:47.738890 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:47.738895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:47.738997 (+ 102us) server_negotiation.cc:407] Connection header received
0928 06:35:47.739148 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:47.739150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:47.739197 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:47.739283 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:47.739788 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.740513 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.741742 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.741980 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.742067 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:47.743443 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:47.743446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:47.743448 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:47.743470 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:47.744948 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:47.745453 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:47.745456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:47.745457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:47.745530 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:47.745829 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:47.745831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:47.745832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:47.746047 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:47.746168 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:47.746492 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 06:35:47.746601 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:35:47.746784 13139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:47.738572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:42115 (local address 127.5.22.2:46881)
0928 06:35:47.738876 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:47.738898 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:47.739037 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:47.739299 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:47.739302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:47.739521 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:47.739654 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:47.739662 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.740658 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.740661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:47.741623 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:47.741633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.742114 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.742120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:47.742233 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:47.742781 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:47.742802 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:47.743299 (+ 497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:47.745064 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:47.745070 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:47.745075 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:47.745356 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:47.745621 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:47.745627 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:47.745632 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:47.745738 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:47.746157 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:47.746160 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:47.746261 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 06:35:47.746433 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 06:35:47.746593 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for oryx/127.5.22.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.22.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.22.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.3.keytab.
Entry for principal oryx/127.5.22.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.3.keytab.
I20260928 06:35:47.770181 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:42115
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.3.keytab
--principal=oryx/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:47.888079 13142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:47.888345 13142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:47.888401 13142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:47.892992 13142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:47.893052 13142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:47.893091 13142 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:35:47.893124 13142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:35:47.898722 13142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.3.keytab
--principal=oryx/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.22.62:42115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:47.899904 13142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:47.900914 13142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:47.910727 13147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:47.910741 13148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:47.910959 13142 server_base.cc:1034] running on GCE node
W20260928 06:35:47.910821 13150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:47.911538 13142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:47.912211 13142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:47.913380 13142 hybrid_clock.cc:648] HybridClock initialized: now 1790577347913362 us; error 31 us; skew 500 ppm
Sep 28 06:35:47 dist-test-slave-j0qf krb5kdc[12745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577347, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:47.916298 13142 init.cc:377] Logged in from keytab as oryx/127.5.22.3@KRBTEST.COM (short username oryx)
I20260928 06:35:47.917274 13142 webserver.cc:460] Webserver started at http://127.5.22.3:33787/ using document root <none> and password file <none>
I20260928 06:35:47.917915 13142 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:47.918004 13142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:47.918233 13142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:47.920620 13142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "41a3cda940e4410a86d18ed8cc01aaa3"
format_stamp: "Formatted at 2026-09-28 06:35:47 on dist-test-slave-j0qf"
server_key: "bf9c5d6196976a3a4064c869d54dabf5"
server_key_iv: "55f0f5ffae25d649a99a22bff1913e43"
server_key_version: "encryptionkey@0"
I20260928 06:35:47.921396 13142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "41a3cda940e4410a86d18ed8cc01aaa3"
format_stamp: "Formatted at 2026-09-28 06:35:47 on dist-test-slave-j0qf"
server_key: "bf9c5d6196976a3a4064c869d54dabf5"
server_key_iv: "55f0f5ffae25d649a99a22bff1913e43"
server_key_version: "encryptionkey@0"
I20260928 06:35:47.925912 13142 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 06:35:47.928328 13157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:47.929726 13142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 06:35:47.929836 13142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "41a3cda940e4410a86d18ed8cc01aaa3"
format_stamp: "Formatted at 2026-09-28 06:35:47 on dist-test-slave-j0qf"
server_key: "bf9c5d6196976a3a4064c869d54dabf5"
server_key_iv: "55f0f5ffae25d649a99a22bff1913e43"
server_key_version: "encryptionkey@0"
I20260928 06:35:47.929934 13142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:47.953670 13142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:47.957002 13142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:47.957167 13142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:47.958774 13142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:47.958828 13142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:47.958917 13142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:47.958957 13142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:35:47 dist-test-slave-j0qf krb5kdc[12745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577347, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM
I20260928 06:35:47.971750 13142 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:43023
I20260928 06:35:47.971932 13272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:43023 every 8 connection(s)
I20260928 06:35:47.972790 13142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:47.974794 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:47.959461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:43705 (local address 127.5.22.62:42115)
0928 06:35:47.959638 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:47.959646 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:47.960603 (+ 957us) server_negotiation.cc:407] Connection header received
0928 06:35:47.961962 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:47.961965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:47.962029 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:47.962132 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:47.963632 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.964412 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.965914 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.966142 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.966254 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:47.970633 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:47.970637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:47.970639 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:47.970665 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:47.972458 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:47.972994 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:47.972998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:47.973000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:47.973112 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:47.973639 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:47.973643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:47.973645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:47.973890 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:47.973974 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:47.974572 (+ 598us) server_negotiation.cc:299] Negotiation successful
0928 06:35:47.974677 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 06:35:47.975630 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:47.959891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:42115 (local address 127.5.22.3:43705)
0928 06:35:47.960409 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:47.960477 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:47.961694 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:47.962422 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:47.962430 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:47.962916 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:47.963433 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:47.963448 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.964732 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.964735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:47.965777 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:47.965787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.966291 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.966295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:47.966744 (+ 449us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:47.967407 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:47.967437 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:47.970386 (+ 2949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:47.972590 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:47.972596 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:47.972611 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:47.972872 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:47.973253 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:47.973256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:47.973258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:47.973421 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:47.974016 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:47.974021 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:47.974256 (+ 235us) client_negotiation.cc:769] Sending connection context
0928 06:35:47.974509 (+ 253us) client_negotiation.cc:240] Negotiation successful
0928 06:35:47.974781 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":117,"threads_started":1}
I20260928 06:35:47.976817 13273 heartbeater.cc:347] Connected to a master server at 127.5.22.62:42115
I20260928 06:35:47.976976 13273 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:47.977489 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 13142
I20260928 06:35:47.977559 13273 heartbeater.cc:511] Master 127.5.22.62:42115 requested a full tablet report, sending...
I20260928 06:35:47.977626 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:47.977948 5208 external_mini_cluster.cc:1428] Setting key 95b6774bbcbd40106a4ee243ff6781df
I20260928 06:35:47.978642 12788 ts_manager.cc:194] Registered new tserver with Master: 41a3cda940e4410a86d18ed8cc01aaa3 (127.5.22.3:43023)
I20260928 06:35:47.979238 12788 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.22.3@KRBTEST.COM'} at 127.5.22.3:43705
I20260928 06:35:47.980643 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:35:47.987270 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:47.979342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:55673 (local address 127.5.22.62:42115)
0928 06:35:47.979453 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:47.979457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:47.979634 (+ 177us) server_negotiation.cc:407] Connection header received
0928 06:35:47.979736 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:47.979739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:47.979786 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:47.979867 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:47.980372 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.981108 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.982165 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.982572 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.982684 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:47.983749 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:47.983755 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:47.983761 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:47.983791 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:47.985476 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:47.985898 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:47.985903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:47.985905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:47.985992 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:47.986437 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:47.986439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:47.986440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:47.986658 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:47.986735 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:47.987077 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 06:35:47.987165 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:35:47.987488 13279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:47.979270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:42115 (local address 127.5.22.3:55673)
0928 06:35:47.979539 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:47.979553 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:47.979641 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:47.979974 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:47.979977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:47.980164 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:47.980257 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:47.980262 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.981261 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.981264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:47.982046 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:47.982054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.982713 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.982716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:47.982796 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:47.983285 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:47.983295 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:47.983612 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:47.985603 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:47.985605 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:47.985607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:47.985796 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:47.986198 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:47.986203 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:47.986208 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:47.986311 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:47.986766 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:47.986772 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:47.986890 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 06:35:47.987026 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 06:35:47.987196 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":119,"threads_started":1}
Sep 28 06:35:47 dist-test-slave-j0qf krb5kdc[12745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577347, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:35:48 dist-test-slave-j0qf krb5kdc[12745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:48 dist-test-slave-j0qf krb5kdc[12745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:35:48.005487 12839 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:47.999365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41318 (local address 127.5.22.62:42115)
0928 06:35:47.999494 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:47.999503 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:47.999515 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:35:47.999550 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:47.999553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:47.999589 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:47.999651 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:48.000397 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.000905 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.001780 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.002009 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.002088 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.005372 (+ 3284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41318: BlockingRecv error: recv got EOF from 127.0.0.1:41318 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 06:35:48 dist-test-slave-j0qf krb5kdc[12745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577347, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM
I20260928 06:35:48.015162 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.007205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41324 (local address 127.5.22.62:42115)
0928 06:35:48.007306 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:48.007309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:48.007537 (+ 228us) server_negotiation.cc:407] Connection header received
0928 06:35:48.007597 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:48.007599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:48.007637 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:48.007700 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:48.008221 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.008736 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.009554 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.009772 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.009845 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.012161 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:48.012164 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:48.012167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:48.012191 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:48.013630 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:48.014035 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:48.014038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:48.014039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:48.014116 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:48.014431 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:48.014434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:48.014435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:48.014653 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:48.014765 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:48.014953 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 06:35:48.015046 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:35:48.018214 12788 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:35:48.021309 12788 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:48.035925 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.030305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.5.22.1:43981)
0928 06:35:48.030691 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:48.030697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:48.030914 (+ 217us) server_negotiation.cc:407] Connection header received
0928 06:35:48.031176 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:48.031184 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:48.031362 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:48.031531 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:48.031964 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.033255 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.034607 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.035286 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.035403 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.035539 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:48.035648 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 06:35:48.035741 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":75,"threads_started":1}
I20260928 06:35:48.036015 13299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.030501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:40931 (local address 127.0.0.1:59646)
0928 06:35:48.030830 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:48.030843 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:48.030936 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:48.031671 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:48.031677 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:48.031699 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:48.031856 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.031862 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.033080 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.033084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.034515 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.034526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.035578 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.035582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.035677 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.035697 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 06:35:48.035777 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 06:35:48.035844 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":65,"spinlock_wait_cycles":896,"thread_start_us":85,"threads_started":1}
I20260928 06:35:48.036195 13295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.029294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:43023 (local address 127.0.0.1:45666)
0928 06:35:48.029735 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:48.029790 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:48.029943 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:48.030723 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:48.030731 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:48.030756 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:48.030896 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.030904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.032269 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.032272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.033906 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.033917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.035261 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.035264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.035372 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.035429 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 06:35:48.035532 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 06:35:48.035658 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":1,"thread_start_us":172,"threads_started":1}
I20260928 06:35:48.035868 13297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.030153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:43981 (local address 127.0.0.1:55672)
0928 06:35:48.030815 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:48.030829 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:48.031018 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:48.031733 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:48.031736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:48.031759 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:48.031840 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.031846 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.033355 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.033358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.034474 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.034483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.035404 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.035408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.035513 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.035536 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 06:35:48.035618 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 06:35:48.035694 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":18,"spinlock_wait_cycles":3200,"thread_start_us":88,"threads_started":1}
I20260928 06:35:48.036369 13296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.029505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45666 (local address 127.5.22.3:43023)
0928 06:35:48.029839 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:48.029853 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:48.029877 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:35:48.030112 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:48.030117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:48.030368 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:48.030545 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:48.031027 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.032138 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.034078 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.034906 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.035006 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.035194 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:48.035814 (+ 620us) server_negotiation.cc:299] Negotiation successful
0928 06:35:48.035914 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":111,"threads_started":1}
I20260928 06:35:48.036885 13300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.030907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59646 (local address 127.5.22.2:40931)
0928 06:35:48.031242 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:48.031253 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:48.031275 (+ 22us) server_negotiation.cc:407] Connection header received
0928 06:35:48.031346 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:48.031350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:48.031525 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:48.031670 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:48.032001 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.032958 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.034653 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.035442 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.036450 (+ 1008us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.036624 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:48.036672 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 06:35:48.036752 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":127,"threads_started":1}
I20260928 06:35:48.039167 13067 tablet_service.cc:1467] Processing CreateTablet for tablet e0844ee5a7dd4d8cbb1b6edf25100c17 (DEFAULT_TABLE table=test-table [id=c5a6276a788646ae95b3403c207d0125]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:48.039199 13207 tablet_service.cc:1467] Processing CreateTablet for tablet e0844ee5a7dd4d8cbb1b6edf25100c17 (DEFAULT_TABLE table=test-table [id=c5a6276a788646ae95b3403c207d0125]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:48.039196 12927 tablet_service.cc:1467] Processing CreateTablet for tablet e0844ee5a7dd4d8cbb1b6edf25100c17 (DEFAULT_TABLE table=test-table [id=c5a6276a788646ae95b3403c207d0125]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:48.040125 13067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0844ee5a7dd4d8cbb1b6edf25100c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:48.040165 13207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0844ee5a7dd4d8cbb1b6edf25100c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:48.040169 12927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0844ee5a7dd4d8cbb1b6edf25100c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:48.046957 13301 tablet_bootstrap.cc:492] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68: Bootstrap starting.
I20260928 06:35:48.047791 13302 tablet_bootstrap.cc:492] T e0844ee5a7dd4d8cbb1b6edf25100c17 P e1c6df1bbd4548d9ab9d6b96a197f4e9: Bootstrap starting.
I20260928 06:35:48.049196 13301 tablet_bootstrap.cc:654] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:48.049433 13302 tablet_bootstrap.cc:654] T e0844ee5a7dd4d8cbb1b6edf25100c17 P e1c6df1bbd4548d9ab9d6b96a197f4e9: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:48.049618 13303 tablet_bootstrap.cc:492] T e0844ee5a7dd4d8cbb1b6edf25100c17 P 41a3cda940e4410a86d18ed8cc01aaa3: Bootstrap starting.
I20260928 06:35:48.050158 13302 log.cc:824] T e0844ee5a7dd4d8cbb1b6edf25100c17 P e1c6df1bbd4548d9ab9d6b96a197f4e9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:48.050254 13301 log.cc:824] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:48.051628 13303 tablet_bootstrap.cc:654] T e0844ee5a7dd4d8cbb1b6edf25100c17 P 41a3cda940e4410a86d18ed8cc01aaa3: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:48.052367 13302 tablet_bootstrap.cc:492] T e0844ee5a7dd4d8cbb1b6edf25100c17 P e1c6df1bbd4548d9ab9d6b96a197f4e9: No bootstrap required, opened a new log
I20260928 06:35:48.052379 13301 tablet_bootstrap.cc:492] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68: No bootstrap required, opened a new log
I20260928 06:35:48.052429 13303 log.cc:824] T e0844ee5a7dd4d8cbb1b6edf25100c17 P 41a3cda940e4410a86d18ed8cc01aaa3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:48.052563 13302 ts_tablet_manager.cc:1397] T e0844ee5a7dd4d8cbb1b6edf25100c17 P e1c6df1bbd4548d9ab9d6b96a197f4e9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 06:35:48.052635 13301 ts_tablet_manager.cc:1397] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 06:35:48.054232 13303 tablet_bootstrap.cc:492] T e0844ee5a7dd4d8cbb1b6edf25100c17 P 41a3cda940e4410a86d18ed8cc01aaa3: No bootstrap required, opened a new log
I20260928 06:35:48.054430 13303 ts_tablet_manager.cc:1397] T e0844ee5a7dd4d8cbb1b6edf25100c17 P 41a3cda940e4410a86d18ed8cc01aaa3: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 06:35:48.056365 13302 raft_consensus.cc:348] T e0844ee5a7dd4d8cbb1b6edf25100c17 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } } peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } }
I20260928 06:35:48.056391 13301 raft_consensus.cc:348] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } } peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } }
I20260928 06:35:48.056598 13302 raft_consensus.cc:374] T e0844ee5a7dd4d8cbb1b6edf25100c17 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:48.056638 13301 raft_consensus.cc:374] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:48.056655 13302 raft_consensus.cc:729] T e0844ee5a7dd4d8cbb1b6edf25100c17 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1c6df1bbd4548d9ab9d6b96a197f4e9, State: Initialized, Role: FOLLOWER
I20260928 06:35:48.056691 13301 raft_consensus.cc:729] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6179c451b334f648ae3355cca58bb68, State: Initialized, Role: FOLLOWER
I20260928 06:35:48.057093 13302 consensus_queue.cc:260] T e0844ee5a7dd4d8cbb1b6edf25100c17 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } } peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } }
I20260928 06:35:48.057106 13301 consensus_queue.cc:260] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } } peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } }
I20260928 06:35:48.057703 12993 heartbeater.cc:503] Master 127.5.22.62:42115 was elected leader, sending a full tablet report...
I20260928 06:35:48.058030 13301 ts_tablet_manager.cc:1428] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 06:35:48.058053 13302 ts_tablet_manager.cc:1428] T e0844ee5a7dd4d8cbb1b6edf25100c17 P e1c6df1bbd4548d9ab9d6b96a197f4e9: Time spent starting tablet: real 0.005s user 0.002s sys 0.005s
I20260928 06:35:48.058116 13303 raft_consensus.cc:348] T e0844ee5a7dd4d8cbb1b6edf25100c17 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } } peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } }
I20260928 06:35:48.058374 13303 raft_consensus.cc:374] T e0844ee5a7dd4d8cbb1b6edf25100c17 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:48.058434 13303 raft_consensus.cc:729] T e0844ee5a7dd4d8cbb1b6edf25100c17 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a3cda940e4410a86d18ed8cc01aaa3, State: Initialized, Role: FOLLOWER
I20260928 06:35:48.058483 13133 heartbeater.cc:503] Master 127.5.22.62:42115 was elected leader, sending a full tablet report...
I20260928 06:35:48.059101 13303 consensus_queue.cc:260] T e0844ee5a7dd4d8cbb1b6edf25100c17 P 41a3cda940e4410a86d18ed8cc01aaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } } peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } }
I20260928 06:35:48.059784 13273 heartbeater.cc:503] Master 127.5.22.62:42115 was elected leader, sending a full tablet report...
I20260928 06:35:48.060026 13303 ts_tablet_manager.cc:1428] T e0844ee5a7dd4d8cbb1b6edf25100c17 P 41a3cda940e4410a86d18ed8cc01aaa3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 06:35:48.075297 13308 raft_consensus.cc:482] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:48.075493 13308 raft_consensus.cc:504] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } } peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } }
I20260928 06:35:48.076640 13308 leader_election.cc:288] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41a3cda940e4410a86d18ed8cc01aaa3 (127.5.22.3:43023), e1c6df1bbd4548d9ab9d6b96a197f4e9 (127.5.22.1:43981)
I20260928 06:35:48.081211 13296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.076942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:40243 (local address 127.5.22.3:43023)
0928 06:35:48.077114 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:48.077119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:48.077140 (+ 21us) server_negotiation.cc:407] Connection header received
0928 06:35:48.077324 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:48.077328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:48.077379 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:48.077450 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:48.077845 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.078778 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.080119 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.080685 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.080792 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.080868 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:48.081047 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 06:35:48.081111 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 06:35:48.081228 13026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.076871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:43023 (local address 127.5.22.2:40243)
0928 06:35:48.077041 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:48.077092 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:48.077188 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:48.077574 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:48.077579 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:48.077599 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:48.077709 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.077716 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.078885 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.078888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.079999 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.080008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.080797 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.080799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.080913 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.080937 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 06:35:48.081017 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 06:35:48.081094 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 06:35:48.081936 13227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0844ee5a7dd4d8cbb1b6edf25100c17" candidate_uuid: "f6179c451b334f648ae3355cca58bb68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" is_pre_election: true
I20260928 06:35:48.082087 13310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.077147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:43981 (local address 127.5.22.2:44431)
0928 06:35:48.077587 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:48.077600 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:48.077720 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:48.077983 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:48.077985 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:48.078004 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:48.078708 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.078718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.079728 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.079730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.080794 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.080803 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.081667 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.081671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.081763 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.081789 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 06:35:48.081870 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 06:35:48.081931 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":115,"threads_started":1}
I20260928 06:35:48.082103 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.077287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:44431 (local address 127.5.22.1:43981)
0928 06:35:48.077422 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:48.077427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:48.077682 (+ 255us) server_negotiation.cc:407] Connection header received
0928 06:35:48.077821 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:48.077824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:48.077874 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:48.077955 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:48.078849 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.079608 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.080904 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.081538 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.081660 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.081713 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:48.081907 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 06:35:48.081990 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 06:35:48.082245 13227 raft_consensus.cc:2454] T e0844ee5a7dd4d8cbb1b6edf25100c17 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6179c451b334f648ae3355cca58bb68 in term 0.
I20260928 06:35:48.082737 13021 leader_election.cc:302] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41a3cda940e4410a86d18ed8cc01aaa3, f6179c451b334f648ae3355cca58bb68; no voters:
I20260928 06:35:48.082804 12947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0844ee5a7dd4d8cbb1b6edf25100c17" candidate_uuid: "f6179c451b334f648ae3355cca58bb68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" is_pre_election: true
I20260928 06:35:48.083035 13308 raft_consensus.cc:2781] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:48.083120 13308 raft_consensus.cc:482] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:48.083096 12947 raft_consensus.cc:2454] T e0844ee5a7dd4d8cbb1b6edf25100c17 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6179c451b334f648ae3355cca58bb68 in term 0.
I20260928 06:35:48.083178 13308 raft_consensus.cc:3037] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:48.084287 13308 raft_consensus.cc:504] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } } peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } }
I20260928 06:35:48.084654 13308 leader_election.cc:288] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68 [CANDIDATE]: Term 1 election: Requested vote from peers 41a3cda940e4410a86d18ed8cc01aaa3 (127.5.22.3:43023), e1c6df1bbd4548d9ab9d6b96a197f4e9 (127.5.22.1:43981)
I20260928 06:35:48.084982 13227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0844ee5a7dd4d8cbb1b6edf25100c17" candidate_uuid: "f6179c451b334f648ae3355cca58bb68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41a3cda940e4410a86d18ed8cc01aaa3"
I20260928 06:35:48.084995 12947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0844ee5a7dd4d8cbb1b6edf25100c17" candidate_uuid: "f6179c451b334f648ae3355cca58bb68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9"
I20260928 06:35:48.085094 13227 raft_consensus.cc:3037] T e0844ee5a7dd4d8cbb1b6edf25100c17 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:48.085129 12947 raft_consensus.cc:3037] T e0844ee5a7dd4d8cbb1b6edf25100c17 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:48.086161 13227 raft_consensus.cc:2454] T e0844ee5a7dd4d8cbb1b6edf25100c17 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6179c451b334f648ae3355cca58bb68 in term 1.
I20260928 06:35:48.086179 12947 raft_consensus.cc:2454] T e0844ee5a7dd4d8cbb1b6edf25100c17 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6179c451b334f648ae3355cca58bb68 in term 1.
I20260928 06:35:48.086433 13021 leader_election.cc:302] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41a3cda940e4410a86d18ed8cc01aaa3, f6179c451b334f648ae3355cca58bb68; no voters:
I20260928 06:35:48.086584 13308 raft_consensus.cc:2781] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:48.086805 13308 raft_consensus.cc:686] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68 [term 1 LEADER]: Becoming Leader. State: Replica: f6179c451b334f648ae3355cca58bb68, State: Running, Role: LEADER
I20260928 06:35:48.087128 13308 consensus_queue.cc:237] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } } peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } }
I20260928 06:35:48.090543 12790 catalog_manager.cc:5495] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6179c451b334f648ae3355cca58bb68 (127.5.22.2). New cstate: current_term: 1 leader_uuid: "f6179c451b334f648ae3355cca58bb68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } health_report { overall_health: HEALTHY } } }
I20260928 06:35:48.115599 13300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.111782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59654 (local address 127.5.22.2:40931)
0928 06:35:48.111931 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:48.111937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:48.111952 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:35:48.112166 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:48.112168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:48.112224 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:48.112364 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:48.112615 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.113149 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.114034 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.114272 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.114442 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.114741 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:48.115186 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:48.115314 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:48.115377 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 06:35:48.115445 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 06:35:48.121644 12947 raft_consensus.cc:1261] T e0844ee5a7dd4d8cbb1b6edf25100c17 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 1 FOLLOWER]: Refusing update from remote peer f6179c451b334f648ae3355cca58bb68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:48.121980 13227 raft_consensus.cc:1261] T e0844ee5a7dd4d8cbb1b6edf25100c17 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 1 FOLLOWER]: Refusing update from remote peer f6179c451b334f648ae3355cca58bb68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:48.122370 13311 consensus_queue.cc:1035] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:48.122632 13308 consensus_queue.cc:1035] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:48.130009 13314 mvcc.cc:205] Tried to move back new op lower bound from 7334204817899241472 to 7334204817767473152. Current Snapshot: MvccSnapshot[applied={T|T < 7334204817899241472}]
I20260928 06:35:48.130263 13315 mvcc.cc:205] Tried to move back new op lower bound from 7334204817899241472 to 7334204817767473152. Current Snapshot: MvccSnapshot[applied={T|T < 7334204817899241472}]
I20260928 06:35:48.131526 13316 mvcc.cc:205] Tried to move back new op lower bound from 7334204817899241472 to 7334204817767473152. Current Snapshot: MvccSnapshot[applied={T|T < 7334204817899241472}]
I20260928 06:35:48.137858 12790 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:48.138039 12790 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:48.141463 12790 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2: Sending DeleteTablet for 3 replicas of tablet e0844ee5a7dd4d8cbb1b6edf25100c17
I20260928 06:35:48.142426 12927 tablet_service.cc:1514] Processing DeleteTablet for tablet e0844ee5a7dd4d8cbb1b6edf25100c17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:48 UTC) from {username='oryx', principal='oryx/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:55672
I20260928 06:35:48.142526 13067 tablet_service.cc:1514] Processing DeleteTablet for tablet e0844ee5a7dd4d8cbb1b6edf25100c17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:48 UTC) from {username='oryx', principal='oryx/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:59646
I20260928 06:35:48.143249 13207 tablet_service.cc:1514] Processing DeleteTablet for tablet e0844ee5a7dd4d8cbb1b6edf25100c17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:48 UTC) from {username='oryx', principal='oryx/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:45666
I20260928 06:35:48.143757 13324 tablet_replica.cc:330] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68: stopping tablet replica
I20260928 06:35:48.144114 13324 raft_consensus.cc:2229] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:35:48.144572 13324 raft_consensus.cc:2258] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:48.144985 13323 tablet_replica.cc:330] T e0844ee5a7dd4d8cbb1b6edf25100c17 P e1c6df1bbd4548d9ab9d6b96a197f4e9: stopping tablet replica
I20260928 06:35:48.145340 13323 raft_consensus.cc:2229] T e0844ee5a7dd4d8cbb1b6edf25100c17 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:48.145373 12790 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:35:48.145818 12790 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:48.147078 13324 ts_tablet_manager.cc:1905] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:48.147156 13325 tablet_replica.cc:330] T e0844ee5a7dd4d8cbb1b6edf25100c17 P 41a3cda940e4410a86d18ed8cc01aaa3: stopping tablet replica
I20260928 06:35:48.147394 13325 raft_consensus.cc:2229] T e0844ee5a7dd4d8cbb1b6edf25100c17 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:48.147727 13325 raft_consensus.cc:2258] T e0844ee5a7dd4d8cbb1b6edf25100c17 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:48.149323 13323 raft_consensus.cc:2258] T e0844ee5a7dd4d8cbb1b6edf25100c17 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:48.149936 13324 ts_tablet_manager.cc:1918] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:48.150035 13324 log.cc:1196] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-1/wal/wals/e0844ee5a7dd4d8cbb1b6edf25100c17
I20260928 06:35:48.150380 13324 ts_tablet_manager.cc:1939] T e0844ee5a7dd4d8cbb1b6edf25100c17 P f6179c451b334f648ae3355cca58bb68: Deleting consensus metadata
I20260928 06:35:48.151211 13323 ts_tablet_manager.cc:1905] T e0844ee5a7dd4d8cbb1b6edf25100c17 P e1c6df1bbd4548d9ab9d6b96a197f4e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:48.151782 12777 catalog_manager.cc:4841] TS f6179c451b334f648ae3355cca58bb68 (127.5.22.2:40931): tablet e0844ee5a7dd4d8cbb1b6edf25100c17 (table test-table [id=c5a6276a788646ae95b3403c207d0125]) successfully deleted
I20260928 06:35:48.151913 13207 tablet_service.cc:1467] Processing CreateTablet for tablet 27b7543cab7e48e89ac72623c1cbec0f (DEFAULT_TABLE table=test-table [id=881b16860beb4a75be9feb00cc8278b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:48.152173 13207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b7543cab7e48e89ac72623c1cbec0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:48.152967 12927 tablet_service.cc:1467] Processing CreateTablet for tablet 27b7543cab7e48e89ac72623c1cbec0f (DEFAULT_TABLE table=test-table [id=881b16860beb4a75be9feb00cc8278b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:48.153205 12927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b7543cab7e48e89ac72623c1cbec0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:48.153709 13325 ts_tablet_manager.cc:1905] T e0844ee5a7dd4d8cbb1b6edf25100c17 P 41a3cda940e4410a86d18ed8cc01aaa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:48.155090 13067 tablet_service.cc:1467] Processing CreateTablet for tablet 27b7543cab7e48e89ac72623c1cbec0f (DEFAULT_TABLE table=test-table [id=881b16860beb4a75be9feb00cc8278b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:48.155404 13067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b7543cab7e48e89ac72623c1cbec0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:48.155503 13323 ts_tablet_manager.cc:1918] T e0844ee5a7dd4d8cbb1b6edf25100c17 P e1c6df1bbd4548d9ab9d6b96a197f4e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:48.155586 13323 log.cc:1196] T e0844ee5a7dd4d8cbb1b6edf25100c17 P e1c6df1bbd4548d9ab9d6b96a197f4e9: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-0/wal/wals/e0844ee5a7dd4d8cbb1b6edf25100c17
I20260928 06:35:48.155911 13323 ts_tablet_manager.cc:1939] T e0844ee5a7dd4d8cbb1b6edf25100c17 P e1c6df1bbd4548d9ab9d6b96a197f4e9: Deleting consensus metadata
I20260928 06:35:48.157660 13301 tablet_bootstrap.cc:492] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68: Bootstrap starting.
I20260928 06:35:48.158640 13301 tablet_bootstrap.cc:654] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:48.159358 12777 catalog_manager.cc:4841] TS e1c6df1bbd4548d9ab9d6b96a197f4e9 (127.5.22.1:43981): tablet e0844ee5a7dd4d8cbb1b6edf25100c17 (table test-table [id=c5a6276a788646ae95b3403c207d0125]) successfully deleted
I20260928 06:35:48.159996 13301 tablet_bootstrap.cc:492] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68: No bootstrap required, opened a new log
I20260928 06:35:48.160084 13301 ts_tablet_manager.cc:1397] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:48.160431 13301 raft_consensus.cc:348] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.160511 13301 raft_consensus.cc:374] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:48.160531 13301 raft_consensus.cc:729] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6179c451b334f648ae3355cca58bb68, State: Initialized, Role: FOLLOWER
I20260928 06:35:48.160661 13301 consensus_queue.cc:260] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.160866 13301 ts_tablet_manager.cc:1428] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:35:48.161757 13303 tablet_bootstrap.cc:492] T 27b7543cab7e48e89ac72623c1cbec0f P 41a3cda940e4410a86d18ed8cc01aaa3: Bootstrap starting.
I20260928 06:35:48.162976 13303 tablet_bootstrap.cc:654] T 27b7543cab7e48e89ac72623c1cbec0f P 41a3cda940e4410a86d18ed8cc01aaa3: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:48.163033 13302 tablet_bootstrap.cc:492] T 27b7543cab7e48e89ac72623c1cbec0f P e1c6df1bbd4548d9ab9d6b96a197f4e9: Bootstrap starting.
I20260928 06:35:48.164004 13302 tablet_bootstrap.cc:654] T 27b7543cab7e48e89ac72623c1cbec0f P e1c6df1bbd4548d9ab9d6b96a197f4e9: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:48.164367 13303 tablet_bootstrap.cc:492] T 27b7543cab7e48e89ac72623c1cbec0f P 41a3cda940e4410a86d18ed8cc01aaa3: No bootstrap required, opened a new log
I20260928 06:35:48.164479 13303 ts_tablet_manager.cc:1397] T 27b7543cab7e48e89ac72623c1cbec0f P 41a3cda940e4410a86d18ed8cc01aaa3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:35:48.164942 13303 raft_consensus.cc:348] T 27b7543cab7e48e89ac72623c1cbec0f P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.165066 13303 raft_consensus.cc:374] T 27b7543cab7e48e89ac72623c1cbec0f P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:48.165107 13303 raft_consensus.cc:729] T 27b7543cab7e48e89ac72623c1cbec0f P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a3cda940e4410a86d18ed8cc01aaa3, State: Initialized, Role: FOLLOWER
I20260928 06:35:48.165251 13303 consensus_queue.cc:260] T 27b7543cab7e48e89ac72623c1cbec0f P 41a3cda940e4410a86d18ed8cc01aaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.165553 13303 ts_tablet_manager.cc:1428] T 27b7543cab7e48e89ac72623c1cbec0f P 41a3cda940e4410a86d18ed8cc01aaa3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:48.165902 13325 ts_tablet_manager.cc:1918] T e0844ee5a7dd4d8cbb1b6edf25100c17 P 41a3cda940e4410a86d18ed8cc01aaa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:48.166050 13325 log.cc:1196] T e0844ee5a7dd4d8cbb1b6edf25100c17 P 41a3cda940e4410a86d18ed8cc01aaa3: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/e0844ee5a7dd4d8cbb1b6edf25100c17
I20260928 06:35:48.166436 13325 ts_tablet_manager.cc:1939] T e0844ee5a7dd4d8cbb1b6edf25100c17 P 41a3cda940e4410a86d18ed8cc01aaa3: Deleting consensus metadata
I20260928 06:35:48.167168 13302 tablet_bootstrap.cc:492] T 27b7543cab7e48e89ac72623c1cbec0f P e1c6df1bbd4548d9ab9d6b96a197f4e9: No bootstrap required, opened a new log
I20260928 06:35:48.167239 13302 ts_tablet_manager.cc:1397] T 27b7543cab7e48e89ac72623c1cbec0f P e1c6df1bbd4548d9ab9d6b96a197f4e9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260928 06:35:48.167765 12777 catalog_manager.cc:4841] TS 41a3cda940e4410a86d18ed8cc01aaa3 (127.5.22.3:43023): tablet e0844ee5a7dd4d8cbb1b6edf25100c17 (table test-table [id=c5a6276a788646ae95b3403c207d0125]) successfully deleted
I20260928 06:35:48.167862 13302 raft_consensus.cc:348] T 27b7543cab7e48e89ac72623c1cbec0f P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.167963 13302 raft_consensus.cc:374] T 27b7543cab7e48e89ac72623c1cbec0f P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:48.167989 13302 raft_consensus.cc:729] T 27b7543cab7e48e89ac72623c1cbec0f P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1c6df1bbd4548d9ab9d6b96a197f4e9, State: Initialized, Role: FOLLOWER
I20260928 06:35:48.168115 13302 consensus_queue.cc:260] T 27b7543cab7e48e89ac72623c1cbec0f P e1c6df1bbd4548d9ab9d6b96a197f4e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.168385 13302 ts_tablet_manager.cc:1428] T 27b7543cab7e48e89ac72623c1cbec0f P e1c6df1bbd4548d9ab9d6b96a197f4e9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260928 06:35:48.223987 13274 tablet.cc:2369] T 27b7543cab7e48e89ac72623c1cbec0f P 41a3cda940e4410a86d18ed8cc01aaa3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:35:48.233307 13134 tablet.cc:2369] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:35:48.264400 12994 tablet.cc:2369] T 27b7543cab7e48e89ac72623c1cbec0f P e1c6df1bbd4548d9ab9d6b96a197f4e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:35:48.371438 13311 raft_consensus.cc:482] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:48.371587 13311 raft_consensus.cc:504] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.371969 13311 leader_election.cc:288] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1c6df1bbd4548d9ab9d6b96a197f4e9 (127.5.22.1:43981), 41a3cda940e4410a86d18ed8cc01aaa3 (127.5.22.3:43023)
I20260928 06:35:48.372654 13227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b7543cab7e48e89ac72623c1cbec0f" candidate_uuid: "f6179c451b334f648ae3355cca58bb68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" is_pre_election: true
I20260928 06:35:48.372689 12947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b7543cab7e48e89ac72623c1cbec0f" candidate_uuid: "f6179c451b334f648ae3355cca58bb68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" is_pre_election: true
I20260928 06:35:48.372831 13227 raft_consensus.cc:2454] T 27b7543cab7e48e89ac72623c1cbec0f P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6179c451b334f648ae3355cca58bb68 in term 0.
I20260928 06:35:48.372860 12947 raft_consensus.cc:2454] T 27b7543cab7e48e89ac72623c1cbec0f P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6179c451b334f648ae3355cca58bb68 in term 0.
I20260928 06:35:48.373178 13021 leader_election.cc:302] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41a3cda940e4410a86d18ed8cc01aaa3, f6179c451b334f648ae3355cca58bb68; no voters:
I20260928 06:35:48.373406 13311 raft_consensus.cc:2781] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:48.373489 13311 raft_consensus.cc:482] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:48.373519 13311 raft_consensus.cc:3037] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:48.374734 13311 raft_consensus.cc:504] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.375063 13311 leader_election.cc:288] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68 [CANDIDATE]: Term 1 election: Requested vote from peers e1c6df1bbd4548d9ab9d6b96a197f4e9 (127.5.22.1:43981), 41a3cda940e4410a86d18ed8cc01aaa3 (127.5.22.3:43023)
I20260928 06:35:48.375448 12947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b7543cab7e48e89ac72623c1cbec0f" candidate_uuid: "f6179c451b334f648ae3355cca58bb68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9"
I20260928 06:35:48.375491 13227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b7543cab7e48e89ac72623c1cbec0f" candidate_uuid: "f6179c451b334f648ae3355cca58bb68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41a3cda940e4410a86d18ed8cc01aaa3"
I20260928 06:35:48.375590 12947 raft_consensus.cc:3037] T 27b7543cab7e48e89ac72623c1cbec0f P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:48.375614 13227 raft_consensus.cc:3037] T 27b7543cab7e48e89ac72623c1cbec0f P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:48.376497 12947 raft_consensus.cc:2454] T 27b7543cab7e48e89ac72623c1cbec0f P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6179c451b334f648ae3355cca58bb68 in term 1.
I20260928 06:35:48.376510 13227 raft_consensus.cc:2454] T 27b7543cab7e48e89ac72623c1cbec0f P 41a3cda940e4410a86d18ed8cc01aaa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6179c451b334f648ae3355cca58bb68 in term 1.
I20260928 06:35:48.376765 13021 leader_election.cc:302] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41a3cda940e4410a86d18ed8cc01aaa3, f6179c451b334f648ae3355cca58bb68; no voters:
I20260928 06:35:48.376982 13311 raft_consensus.cc:2781] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:48.377095 13311 raft_consensus.cc:686] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68 [term 1 LEADER]: Becoming Leader. State: Replica: f6179c451b334f648ae3355cca58bb68, State: Running, Role: LEADER
I20260928 06:35:48.377260 13311 consensus_queue.cc:237] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.378909 12790 catalog_manager.cc:5495] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6179c451b334f648ae3355cca58bb68 (127.5.22.2). New cstate: current_term: 1 leader_uuid: "f6179c451b334f648ae3355cca58bb68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } health_report { overall_health: UNKNOWN } } }
Sep 28 06:35:48 dist-test-slave-j0qf krb5kdc[12745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577347, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.62@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM
I20260928 06:35:48.425277 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.416165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:42115 (local address 127.0.0.1:59584)
0928 06:35:48.416524 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:48.416538 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:48.416678 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:48.416926 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:48.416928 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:48.417400 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:48.417509 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.417516 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.418361 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.418364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.419103 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.419118 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.419620 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.419625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.419729 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.420282 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:48.420295 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:48.421815 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:48.423577 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:48.423587 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:48.423594 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:48.423907 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:48.424192 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:48.424195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:48.424197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:48.424282 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:48.424667 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:48.424675 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:48.424794 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 06:35:48.424908 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 06:35:48.425077 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":123,"threads_started":1}
I20260928 06:35:48.425277 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.416208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.5.22.62:42115)
0928 06:35:48.416366 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:48.416371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:48.416616 (+ 245us) server_negotiation.cc:407] Connection header received
0928 06:35:48.416779 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:48.416784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:48.416828 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:48.416910 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:48.417646 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.418233 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.419286 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.419495 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.419569 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.421963 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:48.421967 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:48.421969 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:48.421988 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:48.423444 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:48.424007 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:48.424012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:48.424014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:48.424090 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:48.424372 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:48.424376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:48.424377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:48.424561 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:48.424668 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:48.424965 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 06:35:48.425134 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":54}
I20260928 06:35:48.429446 12790 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:59584:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 06:35:48.430161 12790 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:48.435597 12927 tablet_service.cc:1467] Processing CreateTablet for tablet fb91cba9acce443a8a1ad0c654f1d6e1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a5475f03c0d34717a8ba65c9a0edeb31]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:35:48.435648 13207 tablet_service.cc:1467] Processing CreateTablet for tablet fb91cba9acce443a8a1ad0c654f1d6e1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a5475f03c0d34717a8ba65c9a0edeb31]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:35:48.435760 13067 tablet_service.cc:1467] Processing CreateTablet for tablet fb91cba9acce443a8a1ad0c654f1d6e1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a5475f03c0d34717a8ba65c9a0edeb31]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:35:48.436000 12927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb91cba9acce443a8a1ad0c654f1d6e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:48.436000 13207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb91cba9acce443a8a1ad0c654f1d6e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:48.436101 13067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb91cba9acce443a8a1ad0c654f1d6e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:48.438653 13301 tablet_bootstrap.cc:492] T fb91cba9acce443a8a1ad0c654f1d6e1 P f6179c451b334f648ae3355cca58bb68: Bootstrap starting.
I20260928 06:35:48.438836 13303 tablet_bootstrap.cc:492] T fb91cba9acce443a8a1ad0c654f1d6e1 P 41a3cda940e4410a86d18ed8cc01aaa3: Bootstrap starting.
I20260928 06:35:48.438844 13302 tablet_bootstrap.cc:492] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9: Bootstrap starting.
I20260928 06:35:48.439633 13301 tablet_bootstrap.cc:654] T fb91cba9acce443a8a1ad0c654f1d6e1 P f6179c451b334f648ae3355cca58bb68: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:48.439776 13303 tablet_bootstrap.cc:654] T fb91cba9acce443a8a1ad0c654f1d6e1 P 41a3cda940e4410a86d18ed8cc01aaa3: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:48.439823 13302 tablet_bootstrap.cc:654] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:48.440871 13301 tablet_bootstrap.cc:492] T fb91cba9acce443a8a1ad0c654f1d6e1 P f6179c451b334f648ae3355cca58bb68: No bootstrap required, opened a new log
I20260928 06:35:48.440923 13303 tablet_bootstrap.cc:492] T fb91cba9acce443a8a1ad0c654f1d6e1 P 41a3cda940e4410a86d18ed8cc01aaa3: No bootstrap required, opened a new log
I20260928 06:35:48.440964 13301 ts_tablet_manager.cc:1397] T fb91cba9acce443a8a1ad0c654f1d6e1 P f6179c451b334f648ae3355cca58bb68: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:48.440986 13303 ts_tablet_manager.cc:1397] T fb91cba9acce443a8a1ad0c654f1d6e1 P 41a3cda940e4410a86d18ed8cc01aaa3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:35:48.441390 13303 raft_consensus.cc:348] T fb91cba9acce443a8a1ad0c654f1d6e1 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.441390 13301 raft_consensus.cc:348] T fb91cba9acce443a8a1ad0c654f1d6e1 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.441510 13303 raft_consensus.cc:374] T fb91cba9acce443a8a1ad0c654f1d6e1 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:48.441510 13301 raft_consensus.cc:374] T fb91cba9acce443a8a1ad0c654f1d6e1 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:48.441546 13301 raft_consensus.cc:729] T fb91cba9acce443a8a1ad0c654f1d6e1 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6179c451b334f648ae3355cca58bb68, State: Initialized, Role: FOLLOWER
I20260928 06:35:48.441546 13303 raft_consensus.cc:729] T fb91cba9acce443a8a1ad0c654f1d6e1 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a3cda940e4410a86d18ed8cc01aaa3, State: Initialized, Role: FOLLOWER
I20260928 06:35:48.441563 13302 tablet_bootstrap.cc:492] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9: No bootstrap required, opened a new log
I20260928 06:35:48.441646 13302 ts_tablet_manager.cc:1397] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 06:35:48.441671 13303 consensus_queue.cc:260] T fb91cba9acce443a8a1ad0c654f1d6e1 P 41a3cda940e4410a86d18ed8cc01aaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.441671 13301 consensus_queue.cc:260] T fb91cba9acce443a8a1ad0c654f1d6e1 P f6179c451b334f648ae3355cca58bb68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.441931 13301 ts_tablet_manager.cc:1428] T fb91cba9acce443a8a1ad0c654f1d6e1 P f6179c451b334f648ae3355cca58bb68: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:48.441931 13303 ts_tablet_manager.cc:1428] T fb91cba9acce443a8a1ad0c654f1d6e1 P 41a3cda940e4410a86d18ed8cc01aaa3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:48.442116 13302 raft_consensus.cc:348] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.442126 13309 tablet_replica.cc:439] T fb91cba9acce443a8a1ad0c654f1d6e1 P 41a3cda940e4410a86d18ed8cc01aaa3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } } }
I20260928 06:35:48.442233 13302 raft_consensus.cc:374] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:48.442260 13309 tablet_replica.cc:442] T fb91cba9acce443a8a1ad0c654f1d6e1 P 41a3cda940e4410a86d18ed8cc01aaa3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:48.442277 13302 raft_consensus.cc:729] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1c6df1bbd4548d9ab9d6b96a197f4e9, State: Initialized, Role: FOLLOWER
I20260928 06:35:48.442397 13311 tablet_replica.cc:439] T fb91cba9acce443a8a1ad0c654f1d6e1 P f6179c451b334f648ae3355cca58bb68: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } } }
I20260928 06:35:48.442430 13302 consensus_queue.cc:260] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.442512 13311 tablet_replica.cc:442] T fb91cba9acce443a8a1ad0c654f1d6e1 P f6179c451b334f648ae3355cca58bb68: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:48.442742 13302 ts_tablet_manager.cc:1428] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:48.442958 13307 tablet_replica.cc:439] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } } }
I20260928 06:35:48.443099 13307 tablet_replica.cc:442] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:48.456025 13307 raft_consensus.cc:482] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:48.456178 13307 raft_consensus.cc:504] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.457620 13307 leader_election.cc:288] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6179c451b334f648ae3355cca58bb68 (127.5.22.2:40931), 41a3cda940e4410a86d18ed8cc01aaa3 (127.5.22.3:43023)
I20260928 06:35:48.461731 13296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.458404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:54795 (local address 127.5.22.3:43023)
0928 06:35:48.458534 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:48.458539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:48.458709 (+ 170us) server_negotiation.cc:407] Connection header received
0928 06:35:48.458851 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:48.458854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:48.458897 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:48.458985 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:48.459246 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.459742 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.460732 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.461142 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.461258 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.461326 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:48.461539 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 06:35:48.461606 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 06:35:48.461781 13340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.458278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:43023 (local address 127.5.22.1:54795)
0928 06:35:48.458610 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:48.458621 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:48.458741 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:48.458995 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:48.458998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:48.459013 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:48.459138 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.459143 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.459856 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.459860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.460613 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.460621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.461262 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.461265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.461370 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.461433 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 06:35:48.461523 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 06:35:48.461598 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":26,"thread_start_us":98,"threads_started":1}
I20260928 06:35:48.461784 13300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.457927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:34985 (local address 127.5.22.2:40931)
0928 06:35:48.458067 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:48.458071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:48.458258 (+ 187us) server_negotiation.cc:407] Connection header received
0928 06:35:48.458485 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:48.458489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:48.458543 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:48.458659 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:48.458976 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.459547 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.460721 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.461168 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.461292 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.461356 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:48.461580 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 06:35:48.461652 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:35:48.461813 13339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.457856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:40931 (local address 127.5.22.1:34985)
0928 06:35:48.458154 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:48.458171 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:48.458277 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:48.458685 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:48.458688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:48.458708 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:48.458855 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.458861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.459669 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.459673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.460576 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.460585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.461296 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.461300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.461439 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.461467 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 06:35:48.461563 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 06:35:48.461638 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":3,"spinlock_wait_cycles":7040,"thread_start_us":129,"threads_started":1}
I20260928 06:35:48.462236 13227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91cba9acce443a8a1ad0c654f1d6e1" candidate_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" is_pre_election: true
I20260928 06:35:48.462399 13227 raft_consensus.cc:2454] T fb91cba9acce443a8a1ad0c654f1d6e1 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1c6df1bbd4548d9ab9d6b96a197f4e9 in term 0.
I20260928 06:35:48.462592 13087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91cba9acce443a8a1ad0c654f1d6e1" candidate_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6179c451b334f648ae3355cca58bb68" is_pre_election: true
I20260928 06:35:48.462849 13087 raft_consensus.cc:2454] T fb91cba9acce443a8a1ad0c654f1d6e1 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1c6df1bbd4548d9ab9d6b96a197f4e9 in term 0.
I20260928 06:35:48.462982 12881 leader_election.cc:302] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41a3cda940e4410a86d18ed8cc01aaa3, e1c6df1bbd4548d9ab9d6b96a197f4e9; no voters:
I20260928 06:35:48.463248 13307 raft_consensus.cc:2781] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:48.463306 13307 raft_consensus.cc:482] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:48.463336 13307 raft_consensus.cc:3037] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:48.464389 13307 raft_consensus.cc:504] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.464742 13307 leader_election.cc:288] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [CANDIDATE]: Term 1 election: Requested vote from peers f6179c451b334f648ae3355cca58bb68 (127.5.22.2:40931), 41a3cda940e4410a86d18ed8cc01aaa3 (127.5.22.3:43023)
I20260928 06:35:48.465080 13087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91cba9acce443a8a1ad0c654f1d6e1" candidate_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6179c451b334f648ae3355cca58bb68"
I20260928 06:35:48.465130 13227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91cba9acce443a8a1ad0c654f1d6e1" candidate_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41a3cda940e4410a86d18ed8cc01aaa3"
I20260928 06:35:48.465231 13087 raft_consensus.cc:3037] T fb91cba9acce443a8a1ad0c654f1d6e1 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:48.465258 13227 raft_consensus.cc:3037] T fb91cba9acce443a8a1ad0c654f1d6e1 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:48.466094 13227 raft_consensus.cc:2454] T fb91cba9acce443a8a1ad0c654f1d6e1 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1c6df1bbd4548d9ab9d6b96a197f4e9 in term 1.
I20260928 06:35:48.466239 13087 raft_consensus.cc:2454] T fb91cba9acce443a8a1ad0c654f1d6e1 P f6179c451b334f648ae3355cca58bb68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1c6df1bbd4548d9ab9d6b96a197f4e9 in term 1.
I20260928 06:35:48.466379 12881 leader_election.cc:302] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41a3cda940e4410a86d18ed8cc01aaa3, e1c6df1bbd4548d9ab9d6b96a197f4e9; no voters:
I20260928 06:35:48.466537 13307 raft_consensus.cc:2781] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:48.466748 13307 raft_consensus.cc:686] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 1 LEADER]: Becoming Leader. State: Replica: e1c6df1bbd4548d9ab9d6b96a197f4e9, State: Running, Role: LEADER
I20260928 06:35:48.466990 13307 consensus_queue.cc:237] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.468212 13341 tablet_replica.cc:439] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9: TxnStatusTablet state changed. Reason: New leader e1c6df1bbd4548d9ab9d6b96a197f4e9. Latest consensus state: current_term: 1 leader_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } } }
I20260928 06:35:48.468335 13341 tablet_replica.cc:442] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:35:48.468709 13343 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 06:35:48.468865 13343 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 06:35:48.469447 12788 catalog_manager.cc:5495] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1c6df1bbd4548d9ab9d6b96a197f4e9 (127.5.22.1). New cstate: current_term: 1 leader_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } health_report { overall_health: UNKNOWN } } }
I20260928 06:35:48.487888 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.484084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54530 (local address 127.5.22.1:43981)
0928 06:35:48.484208 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:48.484212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:48.484388 (+ 176us) server_negotiation.cc:407] Connection header received
0928 06:35:48.484598 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:48.484602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:48.484664 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:48.484762 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:48.485007 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.485514 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.486408 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.486636 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.486745 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.487042 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:48.487461 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:48.487591 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:48.487702 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 06:35:48.487751 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:35:48.487888 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.483966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:43981 (local address 127.0.0.1:54530)
0928 06:35:48.484309 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:48.484321 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:48.484457 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:48.484786 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:48.484789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:48.484798 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:35:48.484904 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.484909 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.485632 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.485637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.486246 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.486254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.486758 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.486762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.486849 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.486889 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:35:48.487574 (+ 685us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:35:48.487590 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 06:35:48.487657 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:35:48.487746 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262}
I20260928 06:35:48.490747 13227 raft_consensus.cc:1261] T fb91cba9acce443a8a1ad0c654f1d6e1 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 1 FOLLOWER]: Refusing update from remote peer e1c6df1bbd4548d9ab9d6b96a197f4e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:48.490854 13087 raft_consensus.cc:1261] T fb91cba9acce443a8a1ad0c654f1d6e1 P f6179c451b334f648ae3355cca58bb68 [term 1 FOLLOWER]: Refusing update from remote peer e1c6df1bbd4548d9ab9d6b96a197f4e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:48.491252 13341 consensus_queue.cc:1035] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:48.491421 13307 consensus_queue.cc:1035] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:48.494051 13311 tablet_replica.cc:439] T fb91cba9acce443a8a1ad0c654f1d6e1 P f6179c451b334f648ae3355cca58bb68: TxnStatusTablet state changed. Reason: New leader e1c6df1bbd4548d9ab9d6b96a197f4e9. Latest consensus state: current_term: 1 leader_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } } }
I20260928 06:35:48.494210 13311 tablet_replica.cc:442] T fb91cba9acce443a8a1ad0c654f1d6e1 P f6179c451b334f648ae3355cca58bb68: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:48.494956 13309 tablet_replica.cc:439] T fb91cba9acce443a8a1ad0c654f1d6e1 P 41a3cda940e4410a86d18ed8cc01aaa3: TxnStatusTablet state changed. Reason: New leader e1c6df1bbd4548d9ab9d6b96a197f4e9. Latest consensus state: current_term: 1 leader_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } } }
I20260928 06:35:48.495106 13309 tablet_replica.cc:442] T fb91cba9acce443a8a1ad0c654f1d6e1 P 41a3cda940e4410a86d18ed8cc01aaa3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:48.496217 13341 tablet_replica.cc:439] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } } }
I20260928 06:35:48.496351 13341 tablet_replica.cc:442] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:35:48.496928 13341 tablet_replica.cc:439] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } } }
I20260928 06:35:48.497048 13341 tablet_replica.cc:442] T fb91cba9acce443a8a1ad0c654f1d6e1 P e1c6df1bbd4548d9ab9d6b96a197f4e9: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:35:48.497658 13311 tablet_replica.cc:439] T fb91cba9acce443a8a1ad0c654f1d6e1 P f6179c451b334f648ae3355cca58bb68: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } } }
I20260928 06:35:48.497764 13311 tablet_replica.cc:442] T fb91cba9acce443a8a1ad0c654f1d6e1 P f6179c451b334f648ae3355cca58bb68: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:48.498739 13309 tablet_replica.cc:439] T fb91cba9acce443a8a1ad0c654f1d6e1 P 41a3cda940e4410a86d18ed8cc01aaa3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } } }
I20260928 06:35:48.498849 13309 tablet_replica.cc:442] T fb91cba9acce443a8a1ad0c654f1d6e1 P 41a3cda940e4410a86d18ed8cc01aaa3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:48.513849 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.509583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:33885 (local address 127.5.22.1:43981)
0928 06:35:48.509751 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:48.509755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:48.510642 (+ 887us) server_negotiation.cc:407] Connection header received
0928 06:35:48.510765 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:48.510767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:48.510812 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:48.510915 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:48.511163 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.511864 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.512755 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.513047 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.513154 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.513424 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:48.513503 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:48.513588 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:48.513682 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 06:35:48.513728 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 06:35:48.513877 13350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.509411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:43981 (local address 127.5.22.2:33885)
0928 06:35:48.510555 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:48.510580 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:48.510685 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:48.510926 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:48.510930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:48.510939 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:35:48.511044 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.511050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.511975 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.511980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.512626 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.512634 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.513157 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.513161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.513245 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.513293 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:35:48.513592 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:35:48.513601 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 06:35:48.513662 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:35:48.513744 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1039,"thread_start_us":107,"threads_started":1}
I20260928 06:35:48.521554 12947 raft_consensus.cc:1261] T 27b7543cab7e48e89ac72623c1cbec0f P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 1 FOLLOWER]: Refusing update from remote peer f6179c451b334f648ae3355cca58bb68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:48.521535 13227 raft_consensus.cc:1261] T 27b7543cab7e48e89ac72623c1cbec0f P 41a3cda940e4410a86d18ed8cc01aaa3 [term 1 FOLLOWER]: Refusing update from remote peer f6179c451b334f648ae3355cca58bb68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:48.521998 13308 consensus_queue.cc:1035] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:48.522276 13311 consensus_queue.cc:1035] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:48.549466 13300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.545755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:43309 (local address 127.5.22.2:40931)
0928 06:35:48.545889 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:48.545894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:48.546129 (+ 235us) server_negotiation.cc:407] Connection header received
0928 06:35:48.546217 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:48.546220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:48.546271 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:48.546387 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:48.546648 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.547244 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.548399 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.548636 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.548744 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.549029 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:48.549110 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:48.549189 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:48.549275 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 06:35:48.549324 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 06:35:48.549490 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.545646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:40931 (local address 127.5.22.1:43309)
0928 06:35:48.546040 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:48.546053 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:48.546147 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:48.546395 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:48.546398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:48.546414 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:35:48.546539 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.546544 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.547359 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.547365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.548281 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.548291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.548744 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.548748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.548847 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.548899 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:35:48.549193 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:35:48.549202 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 06:35:48.549265 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 06:35:48.549337 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":125,"threads_started":1}
I20260928 06:35:48.573586 12787 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:48.573756 12787 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:48.575949 12787 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2: Sending DeleteTablet for 3 replicas of tablet 27b7543cab7e48e89ac72623c1cbec0f
I20260928 06:35:48.576702 13067 tablet_service.cc:1514] Processing DeleteTablet for tablet 27b7543cab7e48e89ac72623c1cbec0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:48 UTC) from {username='oryx', principal='oryx/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:59646
I20260928 06:35:48.576927 13324 tablet_replica.cc:330] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68: stopping tablet replica
I20260928 06:35:48.576894 13207 tablet_service.cc:1514] Processing DeleteTablet for tablet 27b7543cab7e48e89ac72623c1cbec0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:48 UTC) from {username='oryx', principal='oryx/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:45666
I20260928 06:35:48.577059 13324 raft_consensus.cc:2229] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:35:48.577185 13325 tablet_replica.cc:330] T 27b7543cab7e48e89ac72623c1cbec0f P 41a3cda940e4410a86d18ed8cc01aaa3: stopping tablet replica
I20260928 06:35:48.577263 13324 raft_consensus.cc:2258] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:48.577312 13325 raft_consensus.cc:2229] T 27b7543cab7e48e89ac72623c1cbec0f P 41a3cda940e4410a86d18ed8cc01aaa3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:48.577425 13325 raft_consensus.cc:2258] T 27b7543cab7e48e89ac72623c1cbec0f P 41a3cda940e4410a86d18ed8cc01aaa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:48.578127 13324 ts_tablet_manager.cc:1905] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:48.580623 13325 ts_tablet_manager.cc:1905] T 27b7543cab7e48e89ac72623c1cbec0f P 41a3cda940e4410a86d18ed8cc01aaa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:48.581453 13324 ts_tablet_manager.cc:1918] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:35:48.581529 13324 log.cc:1196] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-1/wal/wals/27b7543cab7e48e89ac72623c1cbec0f
I20260928 06:35:48.581818 13324 ts_tablet_manager.cc:1939] T 27b7543cab7e48e89ac72623c1cbec0f P f6179c451b334f648ae3355cca58bb68: Deleting consensus metadata
I20260928 06:35:48.582674 12777 catalog_manager.cc:4841] TS f6179c451b334f648ae3355cca58bb68 (127.5.22.2:40931): tablet 27b7543cab7e48e89ac72623c1cbec0f (table test-table [id=881b16860beb4a75be9feb00cc8278b2]) successfully deleted
I20260928 06:35:48.583467 13325 ts_tablet_manager.cc:1918] T 27b7543cab7e48e89ac72623c1cbec0f P 41a3cda940e4410a86d18ed8cc01aaa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:35:48.583523 13325 log.cc:1196] T 27b7543cab7e48e89ac72623c1cbec0f P 41a3cda940e4410a86d18ed8cc01aaa3: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/27b7543cab7e48e89ac72623c1cbec0f
I20260928 06:35:48.583719 13325 ts_tablet_manager.cc:1939] T 27b7543cab7e48e89ac72623c1cbec0f P 41a3cda940e4410a86d18ed8cc01aaa3: Deleting consensus metadata
I20260928 06:35:48.584635 12777 catalog_manager.cc:4841] TS 41a3cda940e4410a86d18ed8cc01aaa3 (127.5.22.3:43023): tablet 27b7543cab7e48e89ac72623c1cbec0f (table test-table [id=881b16860beb4a75be9feb00cc8278b2]) successfully deleted
I20260928 06:35:48.585492 12927 tablet_service.cc:1514] Processing DeleteTablet for tablet 27b7543cab7e48e89ac72623c1cbec0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:48 UTC) from {username='oryx', principal='oryx/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:55672
I20260928 06:35:48.585975 13323 tablet_replica.cc:330] T 27b7543cab7e48e89ac72623c1cbec0f P e1c6df1bbd4548d9ab9d6b96a197f4e9: stopping tablet replica
I20260928 06:35:48.586093 13323 raft_consensus.cc:2229] T 27b7543cab7e48e89ac72623c1cbec0f P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:48.586202 13323 raft_consensus.cc:2258] T 27b7543cab7e48e89ac72623c1cbec0f P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:48.586993 13323 ts_tablet_manager.cc:1905] T 27b7543cab7e48e89ac72623c1cbec0f P e1c6df1bbd4548d9ab9d6b96a197f4e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:48.589540 13323 ts_tablet_manager.cc:1918] T 27b7543cab7e48e89ac72623c1cbec0f P e1c6df1bbd4548d9ab9d6b96a197f4e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:35:48.589630 13323 log.cc:1196] T 27b7543cab7e48e89ac72623c1cbec0f P e1c6df1bbd4548d9ab9d6b96a197f4e9: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-0/wal/wals/27b7543cab7e48e89ac72623c1cbec0f
I20260928 06:35:48.589869 13323 ts_tablet_manager.cc:1939] T 27b7543cab7e48e89ac72623c1cbec0f P e1c6df1bbd4548d9ab9d6b96a197f4e9: Deleting consensus metadata
I20260928 06:35:48.590533 12777 catalog_manager.cc:4841] TS e1c6df1bbd4548d9ab9d6b96a197f4e9 (127.5.22.1:43981): tablet 27b7543cab7e48e89ac72623c1cbec0f (table test-table [id=881b16860beb4a75be9feb00cc8278b2]) successfully deleted
Sep 28 06:35:48 dist-test-slave-j0qf krb5kdc[12745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:35:48 dist-test-slave-j0qf krb5kdc[12745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:48 dist-test-slave-j0qf krb5kdc[12745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:35:48.607677 12839 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:48.600198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59594 (local address 127.5.22.62:42115)
0928 06:35:48.600357 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:48.600362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:48.600530 (+ 168us) server_negotiation.cc:407] Connection header received
0928 06:35:48.600679 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:48.600682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:48.600728 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:48.600801 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:48.601683 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.602302 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.603598 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.603831 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.603918 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.607539 (+ 3621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59594: BlockingRecv error: recv got EOF from 127.0.0.1:59594 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 06:35:48 dist-test-slave-j0qf krb5kdc[12745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM
I20260928 06:35:48.619112 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.609296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59608 (local address 127.5.22.62:42115)
0928 06:35:48.609438 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:48.609445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:48.609654 (+ 209us) server_negotiation.cc:407] Connection header received
0928 06:35:48.609741 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:48.609744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:48.609793 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:48.609877 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:48.610493 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.611233 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.612124 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.612464 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.612607 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.615190 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:48.615194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:48.615197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:48.615225 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:48.616873 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:48.617531 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:48.617535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:48.617537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:48.617642 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:48.617925 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:48.617928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:48.617930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:48.618160 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:48.618363 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:48.618828 (+ 465us) server_negotiation.cc:299] Negotiation successful
0928 06:35:48.618946 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 06:35:48.621750 12787 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:35:48.622224 12787 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:48.627451 12927 tablet_service.cc:1467] Processing CreateTablet for tablet ce0de2a582ad42849db1c4146bd13b84 (DEFAULT_TABLE table=test-table [id=14a457ccbb0a499fadb53051f4a99416]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:48.627533 13067 tablet_service.cc:1467] Processing CreateTablet for tablet ce0de2a582ad42849db1c4146bd13b84 (DEFAULT_TABLE table=test-table [id=14a457ccbb0a499fadb53051f4a99416]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:48.627781 13067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce0de2a582ad42849db1c4146bd13b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:48.627782 12927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce0de2a582ad42849db1c4146bd13b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:48.630206 13301 tablet_bootstrap.cc:492] T ce0de2a582ad42849db1c4146bd13b84 P f6179c451b334f648ae3355cca58bb68: Bootstrap starting.
I20260928 06:35:48.630452 13302 tablet_bootstrap.cc:492] T ce0de2a582ad42849db1c4146bd13b84 P e1c6df1bbd4548d9ab9d6b96a197f4e9: Bootstrap starting.
I20260928 06:35:48.631208 13301 tablet_bootstrap.cc:654] T ce0de2a582ad42849db1c4146bd13b84 P f6179c451b334f648ae3355cca58bb68: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:48.631829 13302 tablet_bootstrap.cc:654] T ce0de2a582ad42849db1c4146bd13b84 P e1c6df1bbd4548d9ab9d6b96a197f4e9: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:48.632529 13301 tablet_bootstrap.cc:492] T ce0de2a582ad42849db1c4146bd13b84 P f6179c451b334f648ae3355cca58bb68: No bootstrap required, opened a new log
I20260928 06:35:48.632473 13207 tablet_service.cc:1467] Processing CreateTablet for tablet ce0de2a582ad42849db1c4146bd13b84 (DEFAULT_TABLE table=test-table [id=14a457ccbb0a499fadb53051f4a99416]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:48.632597 13301 ts_tablet_manager.cc:1397] T ce0de2a582ad42849db1c4146bd13b84 P f6179c451b334f648ae3355cca58bb68: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:48.632748 13207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce0de2a582ad42849db1c4146bd13b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:48.633033 13301 raft_consensus.cc:348] T ce0de2a582ad42849db1c4146bd13b84 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.633142 13301 raft_consensus.cc:374] T ce0de2a582ad42849db1c4146bd13b84 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:48.633174 13301 raft_consensus.cc:729] T ce0de2a582ad42849db1c4146bd13b84 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6179c451b334f648ae3355cca58bb68, State: Initialized, Role: FOLLOWER
I20260928 06:35:48.633298 13301 consensus_queue.cc:260] T ce0de2a582ad42849db1c4146bd13b84 P f6179c451b334f648ae3355cca58bb68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.633549 13301 ts_tablet_manager.cc:1428] T ce0de2a582ad42849db1c4146bd13b84 P f6179c451b334f648ae3355cca58bb68: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:48.635222 13303 tablet_bootstrap.cc:492] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3: Bootstrap starting.
I20260928 06:35:48.636185 13303 tablet_bootstrap.cc:654] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:48.637312 13302 tablet_bootstrap.cc:492] T ce0de2a582ad42849db1c4146bd13b84 P e1c6df1bbd4548d9ab9d6b96a197f4e9: No bootstrap required, opened a new log
I20260928 06:35:48.637418 13302 ts_tablet_manager.cc:1397] T ce0de2a582ad42849db1c4146bd13b84 P e1c6df1bbd4548d9ab9d6b96a197f4e9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260928 06:35:48.637538 13303 tablet_bootstrap.cc:492] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3: No bootstrap required, opened a new log
I20260928 06:35:48.637632 13303 ts_tablet_manager.cc:1397] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:48.637923 13302 raft_consensus.cc:348] T ce0de2a582ad42849db1c4146bd13b84 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.638056 13302 raft_consensus.cc:374] T ce0de2a582ad42849db1c4146bd13b84 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:48.638096 13302 raft_consensus.cc:729] T ce0de2a582ad42849db1c4146bd13b84 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1c6df1bbd4548d9ab9d6b96a197f4e9, State: Initialized, Role: FOLLOWER
I20260928 06:35:48.638036 13303 raft_consensus.cc:348] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.638144 13303 raft_consensus.cc:374] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:48.638180 13303 raft_consensus.cc:729] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a3cda940e4410a86d18ed8cc01aaa3, State: Initialized, Role: FOLLOWER
I20260928 06:35:48.638271 13302 consensus_queue.cc:260] T ce0de2a582ad42849db1c4146bd13b84 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.638302 13303 consensus_queue.cc:260] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.638635 13303 ts_tablet_manager.cc:1428] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:48.639459 13302 ts_tablet_manager.cc:1428] T ce0de2a582ad42849db1c4146bd13b84 P e1c6df1bbd4548d9ab9d6b96a197f4e9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:48.751976 13309 raft_consensus.cc:482] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:48.752156 13309 raft_consensus.cc:504] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.753226 13309 leader_election.cc:288] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1c6df1bbd4548d9ab9d6b96a197f4e9 (127.5.22.1:43981), f6179c451b334f648ae3355cca58bb68 (127.5.22.2:40931)
I20260928 06:35:48.757911 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.753698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:33423 (local address 127.5.22.1:43981)
0928 06:35:48.753828 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:48.753831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:48.753866 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:35:48.754059 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:48.754063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:48.754126 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:48.754182 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:48.754591 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.755261 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.756731 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.757365 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.757449 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.757511 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:48.757735 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 06:35:48.757790 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:35:48.757953 13372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.753524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:43981 (local address 127.5.22.3:33423)
0928 06:35:48.753789 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:48.753807 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:48.753901 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:48.754309 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:48.754330 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:48.754379 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:48.754476 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.754481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.755372 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.755375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.756587 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.756597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.757502 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.757506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.757602 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.757628 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 06:35:48.757714 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:35:48.757785 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":93,"threads_started":1}
I20260928 06:35:48.758440 12947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0de2a582ad42849db1c4146bd13b84" candidate_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" is_pre_election: true
I20260928 06:35:48.758555 13300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.754031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:52331 (local address 127.5.22.2:40931)
0928 06:35:48.754151 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:48.754156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:48.754245 (+ 89us) server_negotiation.cc:407] Connection header received
0928 06:35:48.754433 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:48.754436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:48.754493 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:48.754584 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:48.755163 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.755818 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.757228 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.757832 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.757926 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.758008 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:48.758382 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 06:35:48.758452 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:35:48.758592 12947 raft_consensus.cc:2454] T ce0de2a582ad42849db1c4146bd13b84 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41a3cda940e4410a86d18ed8cc01aaa3 in term 0.
I20260928 06:35:48.758580 13373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.753908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:40931 (local address 127.5.22.3:52331)
0928 06:35:48.754165 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:48.754181 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:48.754291 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:48.754610 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:48.754612 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:48.754634 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:48.755056 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.755062 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.755970 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.755973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.757101 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:48.757112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.757959 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.758096 (+ 137us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:48.758202 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.758224 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 06:35:48.758308 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 06:35:48.758404 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":65,"threads_started":1}
I20260928 06:35:48.759064 13161 leader_election.cc:302] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41a3cda940e4410a86d18ed8cc01aaa3, e1c6df1bbd4548d9ab9d6b96a197f4e9; no voters:
I20260928 06:35:48.759337 13309 raft_consensus.cc:2781] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:48.759423 13309 raft_consensus.cc:482] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:48.759243 13087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0de2a582ad42849db1c4146bd13b84" candidate_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6179c451b334f648ae3355cca58bb68" is_pre_election: true
I20260928 06:35:48.759474 13309 raft_consensus.cc:3037] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:48.759522 13087 raft_consensus.cc:2454] T ce0de2a582ad42849db1c4146bd13b84 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41a3cda940e4410a86d18ed8cc01aaa3 in term 0.
I20260928 06:35:48.760495 13309 raft_consensus.cc:504] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.760780 13309 leader_election.cc:288] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3 [CANDIDATE]: Term 1 election: Requested vote from peers e1c6df1bbd4548d9ab9d6b96a197f4e9 (127.5.22.1:43981), f6179c451b334f648ae3355cca58bb68 (127.5.22.2:40931)
I20260928 06:35:48.761184 13087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0de2a582ad42849db1c4146bd13b84" candidate_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6179c451b334f648ae3355cca58bb68"
I20260928 06:35:48.761150 12947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0de2a582ad42849db1c4146bd13b84" candidate_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9"
I20260928 06:35:48.761297 13087 raft_consensus.cc:3037] T ce0de2a582ad42849db1c4146bd13b84 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:48.761297 12947 raft_consensus.cc:3037] T ce0de2a582ad42849db1c4146bd13b84 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:48.762434 12947 raft_consensus.cc:2454] T ce0de2a582ad42849db1c4146bd13b84 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41a3cda940e4410a86d18ed8cc01aaa3 in term 1.
I20260928 06:35:48.762434 13087 raft_consensus.cc:2454] T ce0de2a582ad42849db1c4146bd13b84 P f6179c451b334f648ae3355cca58bb68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41a3cda940e4410a86d18ed8cc01aaa3 in term 1.
I20260928 06:35:48.762735 13161 leader_election.cc:302] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41a3cda940e4410a86d18ed8cc01aaa3, f6179c451b334f648ae3355cca58bb68; no voters:
I20260928 06:35:48.762913 13309 raft_consensus.cc:2781] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:48.763077 13309 raft_consensus.cc:686] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 1 LEADER]: Becoming Leader. State: Replica: 41a3cda940e4410a86d18ed8cc01aaa3, State: Running, Role: LEADER
I20260928 06:35:48.763324 13309 consensus_queue.cc:237] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.765642 12787 catalog_manager.cc:5495] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41a3cda940e4410a86d18ed8cc01aaa3 (127.5.22.3). New cstate: current_term: 1 leader_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } health_report { overall_health: HEALTHY } } }
I20260928 06:35:48.787689 13296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.783718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41660 (local address 127.5.22.3:43023)
0928 06:35:48.783876 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:48.783880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:48.783904 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:35:48.784063 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:48.784066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:48.784120 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:48.784220 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:48.784629 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.785138 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.786163 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.786397 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.786479 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.786749 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:48.787217 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:48.787329 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:48.787516 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 06:35:48.787573 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 06:35:48.790692 13087 raft_consensus.cc:1261] T ce0de2a582ad42849db1c4146bd13b84 P f6179c451b334f648ae3355cca58bb68 [term 1 FOLLOWER]: Refusing update from remote peer 41a3cda940e4410a86d18ed8cc01aaa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:48.790692 12947 raft_consensus.cc:1261] T ce0de2a582ad42849db1c4146bd13b84 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 1 FOLLOWER]: Refusing update from remote peer 41a3cda940e4410a86d18ed8cc01aaa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:48.791271 13374 consensus_queue.cc:1035] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:48.791484 13309 consensus_queue.cc:1035] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:48.802942 12790 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:48.803121 12790 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:48.805204 12790 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2: Sending DeleteTablet for 3 replicas of tablet ce0de2a582ad42849db1c4146bd13b84
I20260928 06:35:48.805953 12927 tablet_service.cc:1514] Processing DeleteTablet for tablet ce0de2a582ad42849db1c4146bd13b84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:48 UTC) from {username='oryx', principal='oryx/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:55672
I20260928 06:35:48.806183 13067 tablet_service.cc:1514] Processing DeleteTablet for tablet ce0de2a582ad42849db1c4146bd13b84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:48 UTC) from {username='oryx', principal='oryx/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:59646
I20260928 06:35:48.806458 13323 tablet_replica.cc:330] T ce0de2a582ad42849db1c4146bd13b84 P e1c6df1bbd4548d9ab9d6b96a197f4e9: stopping tablet replica
I20260928 06:35:48.806597 13323 raft_consensus.cc:2229] T ce0de2a582ad42849db1c4146bd13b84 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:48.806741 13323 raft_consensus.cc:2258] T ce0de2a582ad42849db1c4146bd13b84 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:48.806687 12790 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 06:35:48.806933 13324 tablet_replica.cc:330] T ce0de2a582ad42849db1c4146bd13b84 P f6179c451b334f648ae3355cca58bb68: stopping tablet replica
I20260928 06:35:48.807056 13324 raft_consensus.cc:2229] T ce0de2a582ad42849db1c4146bd13b84 P f6179c451b334f648ae3355cca58bb68 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260928 06:35:48.807158 12790 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:48.807194 13324 raft_consensus.cc:2258] T ce0de2a582ad42849db1c4146bd13b84 P f6179c451b334f648ae3355cca58bb68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:48.807691 13207 tablet_service.cc:1514] Processing DeleteTablet for tablet ce0de2a582ad42849db1c4146bd13b84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:48 UTC) from {username='oryx', principal='oryx/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:45666
I20260928 06:35:48.807899 13325 tablet_replica.cc:330] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3: stopping tablet replica
I20260928 06:35:48.807999 13325 raft_consensus.cc:2229] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:35:48.808287 13324 ts_tablet_manager.cc:1905] T ce0de2a582ad42849db1c4146bd13b84 P f6179c451b334f648ae3355cca58bb68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:48.808357 13325 raft_consensus.cc:2258] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:48.809065 13323 ts_tablet_manager.cc:1905] T ce0de2a582ad42849db1c4146bd13b84 P e1c6df1bbd4548d9ab9d6b96a197f4e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:48.810079 13325 ts_tablet_manager.cc:1905] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:48.811074 13324 ts_tablet_manager.cc:1918] T ce0de2a582ad42849db1c4146bd13b84 P f6179c451b334f648ae3355cca58bb68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:48.811141 13324 log.cc:1196] T ce0de2a582ad42849db1c4146bd13b84 P f6179c451b334f648ae3355cca58bb68: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-1/wal/wals/ce0de2a582ad42849db1c4146bd13b84
I20260928 06:35:48.811429 13324 ts_tablet_manager.cc:1939] T ce0de2a582ad42849db1c4146bd13b84 P f6179c451b334f648ae3355cca58bb68: Deleting consensus metadata
I20260928 06:35:48.811723 13323 ts_tablet_manager.cc:1918] T ce0de2a582ad42849db1c4146bd13b84 P e1c6df1bbd4548d9ab9d6b96a197f4e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:48.811791 13323 log.cc:1196] T ce0de2a582ad42849db1c4146bd13b84 P e1c6df1bbd4548d9ab9d6b96a197f4e9: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-0/wal/wals/ce0de2a582ad42849db1c4146bd13b84
I20260928 06:35:48.812064 13323 ts_tablet_manager.cc:1939] T ce0de2a582ad42849db1c4146bd13b84 P e1c6df1bbd4548d9ab9d6b96a197f4e9: Deleting consensus metadata
I20260928 06:35:48.812279 12777 catalog_manager.cc:4841] TS f6179c451b334f648ae3355cca58bb68 (127.5.22.2:40931): tablet ce0de2a582ad42849db1c4146bd13b84 (table test-table [id=14a457ccbb0a499fadb53051f4a99416]) successfully deleted
I20260928 06:35:48.812551 13325 ts_tablet_manager.cc:1918] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:48.812618 13325 log.cc:1196] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/ce0de2a582ad42849db1c4146bd13b84
I20260928 06:35:48.812779 12777 catalog_manager.cc:4841] TS e1c6df1bbd4548d9ab9d6b96a197f4e9 (127.5.22.1:43981): tablet ce0de2a582ad42849db1c4146bd13b84 (table test-table [id=14a457ccbb0a499fadb53051f4a99416]) successfully deleted
I20260928 06:35:48.812857 13325 ts_tablet_manager.cc:1939] T ce0de2a582ad42849db1c4146bd13b84 P 41a3cda940e4410a86d18ed8cc01aaa3: Deleting consensus metadata
I20260928 06:35:48.813598 12777 catalog_manager.cc:4841] TS 41a3cda940e4410a86d18ed8cc01aaa3 (127.5.22.3:43023): tablet ce0de2a582ad42849db1c4146bd13b84 (table test-table [id=14a457ccbb0a499fadb53051f4a99416]) successfully deleted
I20260928 06:35:48.816644 12927 tablet_service.cc:1467] Processing CreateTablet for tablet e59c7563e1c74d7c8c07c0525224d7f5 (DEFAULT_TABLE table=test-table [id=d4ba83029cd2493ea107ab4b8e3f53d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:48.816756 13067 tablet_service.cc:1467] Processing CreateTablet for tablet e59c7563e1c74d7c8c07c0525224d7f5 (DEFAULT_TABLE table=test-table [id=d4ba83029cd2493ea107ab4b8e3f53d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:48.816920 12927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e59c7563e1c74d7c8c07c0525224d7f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:48.816875 13207 tablet_service.cc:1467] Processing CreateTablet for tablet e59c7563e1c74d7c8c07c0525224d7f5 (DEFAULT_TABLE table=test-table [id=d4ba83029cd2493ea107ab4b8e3f53d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:48.817021 13067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e59c7563e1c74d7c8c07c0525224d7f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:48.817126 13207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e59c7563e1c74d7c8c07c0525224d7f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:48.819345 13302 tablet_bootstrap.cc:492] T e59c7563e1c74d7c8c07c0525224d7f5 P e1c6df1bbd4548d9ab9d6b96a197f4e9: Bootstrap starting.
I20260928 06:35:48.819416 13303 tablet_bootstrap.cc:492] T e59c7563e1c74d7c8c07c0525224d7f5 P 41a3cda940e4410a86d18ed8cc01aaa3: Bootstrap starting.
I20260928 06:35:48.819416 13301 tablet_bootstrap.cc:492] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68: Bootstrap starting.
I20260928 06:35:48.820165 13302 tablet_bootstrap.cc:654] T e59c7563e1c74d7c8c07c0525224d7f5 P e1c6df1bbd4548d9ab9d6b96a197f4e9: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:48.820238 13301 tablet_bootstrap.cc:654] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:48.820278 13303 tablet_bootstrap.cc:654] T e59c7563e1c74d7c8c07c0525224d7f5 P 41a3cda940e4410a86d18ed8cc01aaa3: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:48.822216 13301 tablet_bootstrap.cc:492] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68: No bootstrap required, opened a new log
I20260928 06:35:48.822290 13301 ts_tablet_manager.cc:1397] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 06:35:48.822696 13301 raft_consensus.cc:348] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.822793 13301 raft_consensus.cc:374] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:48.822815 13301 raft_consensus.cc:729] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6179c451b334f648ae3355cca58bb68, State: Initialized, Role: FOLLOWER
I20260928 06:35:48.822959 13301 consensus_queue.cc:260] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.823230 13301 ts_tablet_manager.cc:1428] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:35:48.823807 13303 tablet_bootstrap.cc:492] T e59c7563e1c74d7c8c07c0525224d7f5 P 41a3cda940e4410a86d18ed8cc01aaa3: No bootstrap required, opened a new log
I20260928 06:35:48.823833 13302 tablet_bootstrap.cc:492] T e59c7563e1c74d7c8c07c0525224d7f5 P e1c6df1bbd4548d9ab9d6b96a197f4e9: No bootstrap required, opened a new log
I20260928 06:35:48.823895 13303 ts_tablet_manager.cc:1397] T e59c7563e1c74d7c8c07c0525224d7f5 P 41a3cda940e4410a86d18ed8cc01aaa3: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260928 06:35:48.823936 13302 ts_tablet_manager.cc:1397] T e59c7563e1c74d7c8c07c0525224d7f5 P e1c6df1bbd4548d9ab9d6b96a197f4e9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 06:35:48.824288 13302 raft_consensus.cc:348] T e59c7563e1c74d7c8c07c0525224d7f5 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.824342 13303 raft_consensus.cc:348] T e59c7563e1c74d7c8c07c0525224d7f5 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.824404 13302 raft_consensus.cc:374] T e59c7563e1c74d7c8c07c0525224d7f5 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:48.824455 13303 raft_consensus.cc:374] T e59c7563e1c74d7c8c07c0525224d7f5 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:48.824493 13303 raft_consensus.cc:729] T e59c7563e1c74d7c8c07c0525224d7f5 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a3cda940e4410a86d18ed8cc01aaa3, State: Initialized, Role: FOLLOWER
I20260928 06:35:48.824601 13302 raft_consensus.cc:729] T e59c7563e1c74d7c8c07c0525224d7f5 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1c6df1bbd4548d9ab9d6b96a197f4e9, State: Initialized, Role: FOLLOWER
I20260928 06:35:48.824729 13303 consensus_queue.cc:260] T e59c7563e1c74d7c8c07c0525224d7f5 P 41a3cda940e4410a86d18ed8cc01aaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.824754 13302 consensus_queue.cc:260] T e59c7563e1c74d7c8c07c0525224d7f5 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.824996 13303 ts_tablet_manager.cc:1428] T e59c7563e1c74d7c8c07c0525224d7f5 P 41a3cda940e4410a86d18ed8cc01aaa3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:48.825042 13302 ts_tablet_manager.cc:1428] T e59c7563e1c74d7c8c07c0525224d7f5 P e1c6df1bbd4548d9ab9d6b96a197f4e9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:48.964169 13311 raft_consensus.cc:482] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:48.964311 13311 raft_consensus.cc:504] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.964807 13311 leader_election.cc:288] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1c6df1bbd4548d9ab9d6b96a197f4e9 (127.5.22.1:43981), 41a3cda940e4410a86d18ed8cc01aaa3 (127.5.22.3:43023)
I20260928 06:35:48.965427 13227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e59c7563e1c74d7c8c07c0525224d7f5" candidate_uuid: "f6179c451b334f648ae3355cca58bb68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" is_pre_election: true
I20260928 06:35:48.965463 12947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e59c7563e1c74d7c8c07c0525224d7f5" candidate_uuid: "f6179c451b334f648ae3355cca58bb68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" is_pre_election: true
I20260928 06:35:48.965595 13227 raft_consensus.cc:2454] T e59c7563e1c74d7c8c07c0525224d7f5 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6179c451b334f648ae3355cca58bb68 in term 0.
I20260928 06:35:48.965627 12947 raft_consensus.cc:2454] T e59c7563e1c74d7c8c07c0525224d7f5 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6179c451b334f648ae3355cca58bb68 in term 0.
I20260928 06:35:48.965996 13021 leader_election.cc:302] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1c6df1bbd4548d9ab9d6b96a197f4e9, f6179c451b334f648ae3355cca58bb68; no voters:
I20260928 06:35:48.966177 13311 raft_consensus.cc:2781] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:48.966239 13311 raft_consensus.cc:482] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:48.966280 13311 raft_consensus.cc:3037] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:48.967382 13311 raft_consensus.cc:504] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.967790 13311 leader_election.cc:288] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68 [CANDIDATE]: Term 1 election: Requested vote from peers e1c6df1bbd4548d9ab9d6b96a197f4e9 (127.5.22.1:43981), 41a3cda940e4410a86d18ed8cc01aaa3 (127.5.22.3:43023)
I20260928 06:35:48.968137 12947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e59c7563e1c74d7c8c07c0525224d7f5" candidate_uuid: "f6179c451b334f648ae3355cca58bb68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9"
I20260928 06:35:48.968281 12947 raft_consensus.cc:3037] T e59c7563e1c74d7c8c07c0525224d7f5 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:48.968230 13227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e59c7563e1c74d7c8c07c0525224d7f5" candidate_uuid: "f6179c451b334f648ae3355cca58bb68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41a3cda940e4410a86d18ed8cc01aaa3"
I20260928 06:35:48.968376 13227 raft_consensus.cc:3037] T e59c7563e1c74d7c8c07c0525224d7f5 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:48.969692 12947 raft_consensus.cc:2454] T e59c7563e1c74d7c8c07c0525224d7f5 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6179c451b334f648ae3355cca58bb68 in term 1.
I20260928 06:35:48.969777 13227 raft_consensus.cc:2454] T e59c7563e1c74d7c8c07c0525224d7f5 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6179c451b334f648ae3355cca58bb68 in term 1.
I20260928 06:35:48.970079 13021 leader_election.cc:302] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1c6df1bbd4548d9ab9d6b96a197f4e9, f6179c451b334f648ae3355cca58bb68; no voters:
I20260928 06:35:48.970278 13311 raft_consensus.cc:2781] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:48.970407 13311 raft_consensus.cc:686] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68 [term 1 LEADER]: Becoming Leader. State: Replica: f6179c451b334f648ae3355cca58bb68, State: Running, Role: LEADER
I20260928 06:35:48.970571 13311 consensus_queue.cc:237] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } }
I20260928 06:35:48.972630 12790 catalog_manager.cc:5495] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6179c451b334f648ae3355cca58bb68 (127.5.22.2). New cstate: current_term: 1 leader_uuid: "f6179c451b334f648ae3355cca58bb68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6179c451b334f648ae3355cca58bb68" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 } health_report { overall_health: UNKNOWN } } }
I20260928 06:35:48.990078 13300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.986397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39106 (local address 127.5.22.2:40931)
0928 06:35:48.986510 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:48.986517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:48.986532 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:35:48.986693 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:48.986695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:48.986737 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:48.986801 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:48.987067 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.987737 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.988988 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.989205 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.989296 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.989588 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:48.989664 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:48.989739 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:48.989827 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 06:35:48.989875 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:35:48.996745 12947 raft_consensus.cc:1261] T e59c7563e1c74d7c8c07c0525224d7f5 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 1 FOLLOWER]: Refusing update from remote peer f6179c451b334f648ae3355cca58bb68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:48.997282 13311 consensus_queue.cc:1035] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1c6df1bbd4548d9ab9d6b96a197f4e9" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:48.997663 13227 raft_consensus.cc:1261] T e59c7563e1c74d7c8c07c0525224d7f5 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 1 FOLLOWER]: Refusing update from remote peer f6179c451b334f648ae3355cca58bb68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:48.998574 13311 consensus_queue.cc:1035] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41a3cda940e4410a86d18ed8cc01aaa3" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:49.028107 12790 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:49.028290 12790 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:49.030282 12790 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 447d544f4e8c4b0a88a28db7251bbbd2: Sending DeleteTablet for 3 replicas of tablet e59c7563e1c74d7c8c07c0525224d7f5
I20260928 06:35:49.030902 12927 tablet_service.cc:1514] Processing DeleteTablet for tablet e59c7563e1c74d7c8c07c0525224d7f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:49 UTC) from {username='oryx', principal='oryx/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:55672
I20260928 06:35:49.031095 13323 tablet_replica.cc:330] T e59c7563e1c74d7c8c07c0525224d7f5 P e1c6df1bbd4548d9ab9d6b96a197f4e9: stopping tablet replica
I20260928 06:35:49.031213 13323 raft_consensus.cc:2229] T e59c7563e1c74d7c8c07c0525224d7f5 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:49.031343 13323 raft_consensus.cc:2258] T e59c7563e1c74d7c8c07c0525224d7f5 P e1c6df1bbd4548d9ab9d6b96a197f4e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:49.031425 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 12862
I20260928 06:35:49.033485 13207 tablet_service.cc:1514] Processing DeleteTablet for tablet e59c7563e1c74d7c8c07c0525224d7f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:49 UTC) from {username='oryx', principal='oryx/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:45666
I20260928 06:35:49.033689 13325 tablet_replica.cc:330] T e59c7563e1c74d7c8c07c0525224d7f5 P 41a3cda940e4410a86d18ed8cc01aaa3: stopping tablet replica
I20260928 06:35:49.033710 13067 tablet_service.cc:1514] Processing DeleteTablet for tablet e59c7563e1c74d7c8c07c0525224d7f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:49 UTC) from {username='oryx', principal='oryx/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:59646
I20260928 06:35:49.033823 13325 raft_consensus.cc:2229] T e59c7563e1c74d7c8c07c0525224d7f5 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:49.033880 13324 tablet_replica.cc:330] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68: stopping tablet replica
I20260928 06:35:49.033938 13325 raft_consensus.cc:2258] T e59c7563e1c74d7c8c07c0525224d7f5 P 41a3cda940e4410a86d18ed8cc01aaa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:49.033989 13324 raft_consensus.cc:2229] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:35:49.034195 13324 raft_consensus.cc:2258] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:49.034684 13325 ts_tablet_manager.cc:1905] T e59c7563e1c74d7c8c07c0525224d7f5 P 41a3cda940e4410a86d18ed8cc01aaa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:49.034781 13324 ts_tablet_manager.cc:1905] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:49.037118 13324 ts_tablet_manager.cc:1918] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:35:49.037119 13325 ts_tablet_manager.cc:1918] T e59c7563e1c74d7c8c07c0525224d7f5 P 41a3cda940e4410a86d18ed8cc01aaa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:35:49.037187 13324 log.cc:1196] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-1/wal/wals/e59c7563e1c74d7c8c07c0525224d7f5
I20260928 06:35:49.037186 13325 log.cc:1196] T e59c7563e1c74d7c8c07c0525224d7f5 P 41a3cda940e4410a86d18ed8cc01aaa3: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/e59c7563e1c74d7c8c07c0525224d7f5
I20260928 06:35:49.037444 13324 ts_tablet_manager.cc:1939] T e59c7563e1c74d7c8c07c0525224d7f5 P f6179c451b334f648ae3355cca58bb68: Deleting consensus metadata
I20260928 06:35:49.037444 13325 ts_tablet_manager.cc:1939] T e59c7563e1c74d7c8c07c0525224d7f5 P 41a3cda940e4410a86d18ed8cc01aaa3: Deleting consensus metadata
I20260928 06:35:49.038245 12777 catalog_manager.cc:4841] TS f6179c451b334f648ae3355cca58bb68 (127.5.22.2:40931): tablet e59c7563e1c74d7c8c07c0525224d7f5 (table test-table [id=d4ba83029cd2493ea107ab4b8e3f53d7]) successfully deleted
I20260928 06:35:49.038697 12777 catalog_manager.cc:4841] TS 41a3cda940e4410a86d18ed8cc01aaa3 (127.5.22.3:43023): tablet e59c7563e1c74d7c8c07c0525224d7f5 (table test-table [id=d4ba83029cd2493ea107ab4b8e3f53d7]) successfully deleted
W20260928 06:35:49.039741 12777 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.22.1:43981 (error 108)
I20260928 06:35:49.039951 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 13002
I20260928 06:35:49.043110 13389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:49.042702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:43981 (local address 127.0.0.1:54532)
0928 06:35:49.042906 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:49.043023 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.1:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":58,"threads_started":1}
W20260928 06:35:49.043278 12777 catalog_manager.cc:4584] TS e1c6df1bbd4548d9ab9d6b96a197f4e9 (127.5.22.1:43981): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e59c7563e1c74d7c8c07c0525224d7f5: Network error: Client connection negotiation failed: client connection to 127.5.22.1:43981: connect: Connection refused (error 111)
I20260928 06:35:49.048347 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 13142
I20260928 06:35:49.055228 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 12758
2026-09-28T06:35:49Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3305 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:49 dist-test-slave-j0qf krb5kdc[13392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:49 dist-test-slave-j0qf krb5kdc[13392](info): set up 2 sockets
Sep 28 06:35:49 dist-test-slave-j0qf krb5kdc[13392](info): commencing operation
krb5kdc: starting...
W20260928 06:35:50.145761 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:35:50 dist-test-slave-j0qf krb5kdc[13392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:35:50.269563 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:43735
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:35507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:43735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:50.379616 13405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:50.379878 13405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:50.379917 13405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:50.383854 13405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:50.383910 13405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:50.383929 13405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:50.383946 13405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:50.383961 13405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:50.389142 13405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:43735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:43735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:50.390067 13405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:50.391045 13405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:50.399477 13411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:50.399528 13410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:50.399602 13405 server_base.cc:1034] running on GCE node
W20260928 06:35:50.399506 13413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:50.400252 13405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:50.401337 13405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:50.402539 13405 hybrid_clock.cc:648] HybridClock initialized: now 1790577350402513 us; error 38 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T06:35:50Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1360 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:50 dist-test-slave-j0qf krb5kdc[13418](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:50 dist-test-slave-j0qf krb5kdc[13418](info): set up 2 sockets
Sep 28 06:35:50 dist-test-slave-j0qf krb5kdc[13418](info): commencing operation
krb5kdc: starting...
W20260928 06:35:51.457499 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:35:51 dist-test-slave-j0qf krb5kdc[13418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:35:51.585764 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:43529
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:40263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:43529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:51.696518 13431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:51.696818 13431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:51.696892 13431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:51.701030 13431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:51.701119 13431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:51.701155 13431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:51.701192 13431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:51.701227 13431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:51.706532 13431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:40263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:43529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:43529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:51.707585 13431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:51.708606 13431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:51.717034 13437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:51.717034 13439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:51.717118 13436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:51.717389 13431 server_base.cc:1034] running on GCE node
I20260928 06:35:51.717983 13431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:51.719172 13431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:51.720338 13431 hybrid_clock.cc:648] HybridClock initialized: now 1790577351720319 us; error 34 us; skew 500 ppm
Sep 28 06:35:51 dist-test-slave-j0qf krb5kdc[13418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577351, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:51.723140 13431 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:35:51.723949 13431 webserver.cc:460] Webserver started at http://127.5.22.62:41739/ using document root <none> and password file <none>
I20260928 06:35:51.724502 13431 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:51.724563 13431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:51.724731 13431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:51.726758 13431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "8a40a2f78dd34f83b2dc371f81403400"
format_stamp: "Formatted at 2026-09-28 06:35:51 on dist-test-slave-j0qf"
server_key: "d6842f1391e046c10148be295c3dd791"
server_key_iv: "9695815b2f13f09876b879290fcb4731"
server_key_version: "encryptionkey@0"
I20260928 06:35:51.727272 13431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "8a40a2f78dd34f83b2dc371f81403400"
format_stamp: "Formatted at 2026-09-28 06:35:51 on dist-test-slave-j0qf"
server_key: "d6842f1391e046c10148be295c3dd791"
server_key_iv: "9695815b2f13f09876b879290fcb4731"
server_key_version: "encryptionkey@0"
I20260928 06:35:51.730958 13431 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:51.733304 13446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:51.734568 13431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:35:51.734706 13431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "8a40a2f78dd34f83b2dc371f81403400"
format_stamp: "Formatted at 2026-09-28 06:35:51 on dist-test-slave-j0qf"
server_key: "d6842f1391e046c10148be295c3dd791"
server_key_iv: "9695815b2f13f09876b879290fcb4731"
server_key_version: "encryptionkey@0"
I20260928 06:35:51.734810 13431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:51.756134 13431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:51.759119 13431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:51.759321 13431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:51.766134 13431 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:43529
I20260928 06:35:51.766222 13498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:43529 every 8 connection(s)
I20260928 06:35:51.767113 13431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:51.769785 13499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:51.772148 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 13431
I20260928 06:35:51.772287 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:35:51.772625 5208 external_mini_cluster.cc:1428] Setting key fcae0539bbca6ceb2b6294037617fdbb
I20260928 06:35:51.776405 13499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400: Bootstrap starting.
I20260928 06:35:51.779057 13499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:51.779861 13499 log.cc:824] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400: Log is configured to *not* fsync() on all Append() calls
Sep 28 06:35:51 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:51.782151 13499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400: No bootstrap required, opened a new log
I20260928 06:35:51.786588 13499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a40a2f78dd34f83b2dc371f81403400" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43529 } }
I20260928 06:35:51.786819 13499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:51.786880 13499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a40a2f78dd34f83b2dc371f81403400, State: Initialized, Role: FOLLOWER
I20260928 06:35:51.787379 13499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a40a2f78dd34f83b2dc371f81403400" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43529 } }
I20260928 06:35:51.787508 13499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:51.787591 13502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:51.774069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39816 (local address 127.5.22.62:43529)
0928 06:35:51.774507 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:51.774522 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:51.774558 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:35:51.775403 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:51.775422 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:51.775779 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:51.776061 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:51.776995 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.778035 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.779312 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.779781 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.779899 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:51.782754 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:51.782759 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:51.782772 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:51.782798 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:51.784454 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:51.784951 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:51.784956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:51.784960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:51.785090 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:51.785383 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:51.785385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:51.785386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:51.785760 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:51.785922 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:51.786226 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 06:35:51.786608 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":137,"threads_started":1}
I20260928 06:35:51.787604 13499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:51.787773 13499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:51.788751 13499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a40a2f78dd34f83b2dc371f81403400" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43529 } }
I20260928 06:35:51.789113 13499 leader_election.cc:302] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a40a2f78dd34f83b2dc371f81403400; no voters:
I20260928 06:35:51.789448 13499 leader_election.cc:288] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:51.789575 13504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:51.789801 13504 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [term 1 LEADER]: Becoming Leader. State: Replica: 8a40a2f78dd34f83b2dc371f81403400, State: Running, Role: LEADER
I20260928 06:35:51.790141 13504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a40a2f78dd34f83b2dc371f81403400" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43529 } }
I20260928 06:35:51.790745 13499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:51.791275 13504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a40a2f78dd34f83b2dc371f81403400. Latest consensus state: current_term: 1 leader_uuid: "8a40a2f78dd34f83b2dc371f81403400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a40a2f78dd34f83b2dc371f81403400" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43529 } } }
I20260928 06:35:51.791397 13504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:51.791788 13508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:51.792119 13505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a40a2f78dd34f83b2dc371f81403400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a40a2f78dd34f83b2dc371f81403400" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43529 } } }
I20260928 06:35:51.792223 13505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:51.795161 13508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:51.800359 13508 catalog_manager.cc:1347] Generated new cluster ID: 12a8c65d49b8423296e58fa9acd632e0
I20260928 06:35:51.800432 13508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:51.821234 13508 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:51.822377 13508 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:51.828716 13508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400: Generated new TSK 0
I20260928 06:35:51.829466 13508 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.22.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
I20260928 06:35:51.861662 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:43529
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:40263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:51.979580 13525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:51.979862 13525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:51.979908 13525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:51.983879 13525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:51.983935 13525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:51.983995 13525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:35:51.989275 13525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:40263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:43529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:51.990288 13525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:51.991407 13525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:52.001298 13531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:52.001343 13533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:52.001310 13530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:52.001586 13525 server_base.cc:1034] running on GCE node
I20260928 06:35:52.002025 13525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:52.002792 13525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:52.003978 13525 hybrid_clock.cc:648] HybridClock initialized: now 1790577352003958 us; error 32 us; skew 500 ppm
Sep 28 06:35:52 dist-test-slave-j0qf krb5kdc[13418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577352, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:52.007092 13525 init.cc:377] Logged in from keytab as kudu/127.5.22.1@KRBTEST.COM (short username kudu)
I20260928 06:35:52.008029 13525 webserver.cc:460] Webserver started at http://127.5.22.1:34845/ using document root <none> and password file <none>
I20260928 06:35:52.008608 13525 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:52.008673 13525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:52.008846 13525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:52.011024 13525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "3f3a026dea584e3a87cf81ba3a47643e"
format_stamp: "Formatted at 2026-09-28 06:35:52 on dist-test-slave-j0qf"
server_key: "2663ddabe4c555bebda60ffa8f789c30"
server_key_iv: "9cd69744f620a904dedcd7c8dd42de42"
server_key_version: "encryptionkey@0"
I20260928 06:35:52.011559 13525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "3f3a026dea584e3a87cf81ba3a47643e"
format_stamp: "Formatted at 2026-09-28 06:35:52 on dist-test-slave-j0qf"
server_key: "2663ddabe4c555bebda60ffa8f789c30"
server_key_iv: "9cd69744f620a904dedcd7c8dd42de42"
server_key_version: "encryptionkey@0"
I20260928 06:35:52.015340 13525 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 06:35:52.017716 13540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:52.019057 13525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:52.019163 13525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "3f3a026dea584e3a87cf81ba3a47643e"
format_stamp: "Formatted at 2026-09-28 06:35:52 on dist-test-slave-j0qf"
server_key: "2663ddabe4c555bebda60ffa8f789c30"
server_key_iv: "9cd69744f620a904dedcd7c8dd42de42"
server_key_version: "encryptionkey@0"
I20260928 06:35:52.019253 13525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:52.035840 13525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:52.039705 13525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:52.039863 13525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:52.040479 13525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:52.041265 13525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:52.041311 13525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:52.041349 13525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:52.041371 13525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:52.050937 13525 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:36195
I20260928 06:35:52.051012 13653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:36195 every 8 connection(s)
I20260928 06:35:52.051905 13525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:52.058728 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 13525
I20260928 06:35:52.058871 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:52.059185 5208 external_mini_cluster.cc:1428] Setting key 0c49f781ceef7f94978c25d0a552b61a
Sep 28 06:35:52 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577352, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:52.067003 13502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:52.053932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:44445 (local address 127.5.22.62:43529)
0928 06:35:52.054108 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:52.054111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:52.055275 (+ 1164us) server_negotiation.cc:407] Connection header received
0928 06:35:52.056207 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:52.056209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:52.056259 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:52.056342 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:52.057475 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.058016 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.059661 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.059993 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.060131 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:52.063019 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:52.063022 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:52.063024 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:52.063047 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:52.064561 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:52.065251 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:52.065254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:52.065255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:52.065333 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:52.065767 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:52.065770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:52.065771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:52.065957 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:52.066033 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:52.066767 (+ 734us) server_negotiation.cc:299] Negotiation successful
0928 06:35:52.066882 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 06:35:52.068235 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:52.054392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.1:44445)
0928 06:35:52.055079 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:52.055144 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:52.055958 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:52.056491 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:52.056498 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:52.056845 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:52.057342 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:52.057356 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.058139 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.058141 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:52.059509 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:52.059533 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.060157 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.060160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:52.060297 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:52.060920 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:52.060938 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:52.062822 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:52.064720 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:52.064727 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:52.064746 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:52.065135 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:52.065430 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:52.065437 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:52.065442 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:52.065649 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:52.066078 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:52.066088 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:52.066484 (+ 396us) client_negotiation.cc:769] Sending connection context
0928 06:35:52.066718 (+ 234us) client_negotiation.cc:240] Negotiation successful
0928 06:35:52.067107 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for kudu/127.5.22.2@KRBTEST.COM; defaulting to no policy
I20260928 06:35:52.069581 13654 heartbeater.cc:347] Connected to a master server at 127.5.22.62:43529
I20260928 06:35:52.069831 13654 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:52.070304 13654 heartbeater.cc:511] Master 127.5.22.62:43529 requested a full tablet report, sending...
I20260928 06:35:52.072006 13463 ts_manager.cc:194] Registered new tserver with Master: 3f3a026dea584e3a87cf81ba3a47643e (127.5.22.1:36195)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.2@KRBTEST.COM" created.
I20260928 06:35:52.073668 13463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.1@KRBTEST.COM'} at 127.5.22.1:44445
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
I20260928 06:35:52.089381 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:43529
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:40263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:52.203274 13659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:52.203583 13659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:52.203657 13659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:52.207656 13659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:52.207734 13659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:52.207827 13659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:35:52.213006 13659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:40263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:43529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:52.214038 13659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:52.215073 13659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:52.225030 13665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:52.225073 13667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:52.225234 13659 server_base.cc:1034] running on GCE node
W20260928 06:35:52.225368 13664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:52.225780 13659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:52.226473 13659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:52.227640 13659 hybrid_clock.cc:648] HybridClock initialized: now 1790577352227642 us; error 43 us; skew 500 ppm
Sep 28 06:35:52 dist-test-slave-j0qf krb5kdc[13418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577352, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:52.230302 13659 init.cc:377] Logged in from keytab as kudu/127.5.22.2@KRBTEST.COM (short username kudu)
I20260928 06:35:52.231195 13659 webserver.cc:460] Webserver started at http://127.5.22.2:45655/ using document root <none> and password file <none>
I20260928 06:35:52.231739 13659 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:52.231830 13659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:52.232076 13659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:52.234241 13659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "da713a92dad743729f632197bbece7ca"
format_stamp: "Formatted at 2026-09-28 06:35:52 on dist-test-slave-j0qf"
server_key: "bc7b6d859ba4323e1088f63b1c1b4517"
server_key_iv: "1806e99835b2a3029b06d647caf2de3f"
server_key_version: "encryptionkey@0"
I20260928 06:35:52.234880 13659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "da713a92dad743729f632197bbece7ca"
format_stamp: "Formatted at 2026-09-28 06:35:52 on dist-test-slave-j0qf"
server_key: "bc7b6d859ba4323e1088f63b1c1b4517"
server_key_iv: "1806e99835b2a3029b06d647caf2de3f"
server_key_version: "encryptionkey@0"
I20260928 06:35:52.238772 13659 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 06:35:52.241456 13674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:52.242789 13659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:35:52.242936 13659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "da713a92dad743729f632197bbece7ca"
format_stamp: "Formatted at 2026-09-28 06:35:52 on dist-test-slave-j0qf"
server_key: "bc7b6d859ba4323e1088f63b1c1b4517"
server_key_iv: "1806e99835b2a3029b06d647caf2de3f"
server_key_version: "encryptionkey@0"
I20260928 06:35:52.243067 13659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:52.253782 13659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:52.256548 13659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:52.256745 13659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:52.257323 13659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:52.258147 13659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:52.258219 13659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:52.258291 13659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:52.258354 13659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:52.267378 13659 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:42063
I20260928 06:35:52.267408 13787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:42063 every 8 connection(s)
I20260928 06:35:52.268301 13659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:52.276472 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 13659
I20260928 06:35:52.276629 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:52.276957 5208 external_mini_cluster.cc:1428] Setting key 965147afb18e18143aa2dc1136316f3d
Sep 28 06:35:52 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577352, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:52.281519 13502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:52.270088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:52187 (local address 127.5.22.62:43529)
0928 06:35:52.270229 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:52.270233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:52.271257 (+ 1024us) server_negotiation.cc:407] Connection header received
0928 06:35:52.272076 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:52.272079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:52.272122 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:52.272195 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:52.273271 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.273822 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.274988 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.275200 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.275281 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:52.277792 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:52.277795 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:52.277797 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:52.277820 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:52.279386 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:52.279968 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:52.279972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:52.279974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:52.280058 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:52.280388 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:52.280393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:52.280394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:52.280573 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:52.280679 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:52.281300 (+ 621us) server_negotiation.cc:299] Negotiation successful
0928 06:35:52.281402 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 06:35:52.282712 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:52.270480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.2:52187)
0928 06:35:52.271119 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:52.271160 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:52.271901 (+ 741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:52.272322 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:52.272329 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:52.272678 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:52.273144 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:52.273157 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.273933 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.273936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:52.274882 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:52.274892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.275327 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.275329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:52.275470 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:52.275991 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:52.276006 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:52.277615 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:52.279577 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:52.279583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:52.279597 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:52.279845 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:52.280150 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:52.280155 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:52.280157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:52.280294 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:52.280677 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:52.280683 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:52.281032 (+ 349us) client_negotiation.cc:769] Sending connection context
0928 06:35:52.281243 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 06:35:52.281591 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":112,"threads_started":1}
I20260928 06:35:52.283962 13788 heartbeater.cc:347] Connected to a master server at 127.5.22.62:43529
I20260928 06:35:52.284215 13788 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:52.284765 13788 heartbeater.cc:511] Master 127.5.22.62:43529 requested a full tablet report, sending...
I20260928 06:35:52.285832 13463 ts_manager.cc:194] Registered new tserver with Master: da713a92dad743729f632197bbece7ca (127.5.22.2:42063)
I20260928 06:35:52.286501 13463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.2@KRBTEST.COM'} at 127.5.22.2:52187
WARNING: no policy specified for kudu/127.5.22.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
I20260928 06:35:52.305444 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:43529
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:40263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:52.421087 13793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:52.421342 13793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:52.421386 13793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:52.425724 13793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:52.425781 13793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:52.425868 13793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:35:52.431118 13793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:40263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:43529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:52.432145 13793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:52.433182 13793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:52.444929 13801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:52.444960 13798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:52.445021 13799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:52.445241 13793 server_base.cc:1034] running on GCE node
I20260928 06:35:52.445789 13793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:52.446573 13793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:52.447755 13793 hybrid_clock.cc:648] HybridClock initialized: now 1790577352447700 us; error 77 us; skew 500 ppm
Sep 28 06:35:52 dist-test-slave-j0qf krb5kdc[13418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577352, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:52.450611 13793 init.cc:377] Logged in from keytab as kudu/127.5.22.3@KRBTEST.COM (short username kudu)
I20260928 06:35:52.451546 13793 webserver.cc:460] Webserver started at http://127.5.22.3:38129/ using document root <none> and password file <none>
I20260928 06:35:52.452111 13793 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:52.452172 13793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:52.452394 13793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:52.454501 13793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "4cd1cf0ab5904e15966e38903de50c7e"
format_stamp: "Formatted at 2026-09-28 06:35:52 on dist-test-slave-j0qf"
server_key: "b0ff9017c98506701f33e95ca4608045"
server_key_iv: "2ca5000fc7bddd934c24eee63bd5a488"
server_key_version: "encryptionkey@0"
I20260928 06:35:52.455060 13793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "4cd1cf0ab5904e15966e38903de50c7e"
format_stamp: "Formatted at 2026-09-28 06:35:52 on dist-test-slave-j0qf"
server_key: "b0ff9017c98506701f33e95ca4608045"
server_key_iv: "2ca5000fc7bddd934c24eee63bd5a488"
server_key_version: "encryptionkey@0"
I20260928 06:35:52.458817 13793 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:35:52.461319 13808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:52.462848 13793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:52.463002 13793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "4cd1cf0ab5904e15966e38903de50c7e"
format_stamp: "Formatted at 2026-09-28 06:35:52 on dist-test-slave-j0qf"
server_key: "b0ff9017c98506701f33e95ca4608045"
server_key_iv: "2ca5000fc7bddd934c24eee63bd5a488"
server_key_version: "encryptionkey@0"
I20260928 06:35:52.463119 13793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:52.491307 13793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:52.494823 13793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:52.495043 13793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:52.495714 13793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:52.496546 13793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:52.496618 13793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:52.496691 13793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:52.496735 13793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:52.507773 13793 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:40555
I20260928 06:35:52.507797 13921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:40555 every 8 connection(s)
I20260928 06:35:52.508773 13793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:52.515466 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 13793
I20260928 06:35:52.515631 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:52.515925 5208 external_mini_cluster.cc:1428] Setting key 9ad5ba3de3af2c5a3519c3768e4aaa6f
Sep 28 06:35:52 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577352, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:35:52.526290 13502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:52.511165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:40827 (local address 127.5.22.62:43529)
0928 06:35:52.511402 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:52.511410 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:52.512657 (+ 1247us) server_negotiation.cc:407] Connection header received
0928 06:35:52.514044 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:52.514047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:52.514095 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:52.514173 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:52.516074 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.517060 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.519097 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.519371 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.519460 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:52.522548 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:52.522552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:52.522554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:52.522588 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:52.524163 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:52.524694 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:52.524703 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:52.524704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:52.524779 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:52.525146 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:52.525149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:52.525150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:52.525344 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:52.525457 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:52.525983 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 06:35:52.526117 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 06:35:52.527146 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:52.511541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.3:40827)
0928 06:35:52.512449 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:52.512515 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:52.513799 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:52.514433 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:52.514445 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:52.515025 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:52.515852 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:52.515875 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.517244 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.517250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:52.518947 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:52.518960 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.519536 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.519543 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:52.519796 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:52.520490 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:52.520513 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:52.522384 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:52.524283 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:52.524288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:52.524305 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:52.524566 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:52.524874 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:52.524879 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:52.524881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:52.525025 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:52.525465 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:52.525474 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:52.525737 (+ 263us) client_negotiation.cc:769] Sending connection context
0928 06:35:52.525925 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 06:35:52.526258 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":153,"threads_started":1}
I20260928 06:35:52.528396 13922 heartbeater.cc:347] Connected to a master server at 127.5.22.62:43529
I20260928 06:35:52.528668 13922 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:52.529304 13922 heartbeater.cc:511] Master 127.5.22.62:43529 requested a full tablet report, sending...
I20260928 06:35:52.530517 13463 ts_manager.cc:194] Registered new tserver with Master: 4cd1cf0ab5904e15966e38903de50c7e (127.5.22.3:40555)
I20260928 06:35:52.531240 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:35:52.531474 13463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.3@KRBTEST.COM'} at 127.5.22.3:40827
WARNING: no policy specified for oryx/127.5.22.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.22.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.22.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.1.keytab.
Entry for principal oryx/127.5.22.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.1.keytab.
WARNING: no policy specified for oryx/127.5.22.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.22.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.22.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.2.keytab.
Entry for principal oryx/127.5.22.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.2.keytab.
WARNING: no policy specified for oryx/127.5.22.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.22.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.22.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.3.keytab.
Entry for principal oryx/127.5.22.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.3.keytab.
I20260928 06:35:52.614707 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 13525
I20260928 06:35:52.620748 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 13659
I20260928 06:35:52.626564 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 13793
I20260928 06:35:52.631882 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 13431
I20260928 06:35:52.639182 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:43529
--webserver_interface=127.5.22.62
--webserver_port=41739
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:40263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:43529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:52.754177 13931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:52.754518 13931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:52.754561 13931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:52.758580 13931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:52.758636 13931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:52.758656 13931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:52.758672 13931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:52.758687 13931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:52.764317 13931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:40263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:43529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:43529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=41739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:52.765916 13931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:52.767162 13931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:52.776460 13937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:52.776505 13936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:52.776505 13939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:52.776662 13931 server_base.cc:1034] running on GCE node
I20260928 06:35:52.777279 13931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:52.778404 13931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:52.779587 13931 hybrid_clock.cc:648] HybridClock initialized: now 1790577352779561 us; error 58 us; skew 500 ppm
Sep 28 06:35:52 dist-test-slave-j0qf krb5kdc[13418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577352, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:52.782459 13931 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:35:52.783420 13931 webserver.cc:460] Webserver started at http://127.5.22.62:41739/ using document root <none> and password file <none>
I20260928 06:35:52.784003 13931 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:52.784114 13931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:52.788067 13931 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:35:52.790387 13946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:52.791862 13931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:35:52.792122 13931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "8a40a2f78dd34f83b2dc371f81403400"
format_stamp: "Formatted at 2026-09-28 06:35:51 on dist-test-slave-j0qf"
server_key: "d6842f1391e046c10148be295c3dd791"
server_key_iv: "9695815b2f13f09876b879290fcb4731"
server_key_version: "encryptionkey@0"
I20260928 06:35:52.792799 13931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:52.802279 13931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:52.805444 13931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:52.805646 13931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:52.813552 13931 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:43529
I20260928 06:35:52.813577 13998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:43529 every 8 connection(s)
I20260928 06:35:52.814704 13931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:52.815835 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 13931
I20260928 06:35:52.816356 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:36195
--local_ip_for_outbound_sockets=127.5.22.1
--tserver_master_addrs=127.5.22.62:43529
--webserver_port=34845
--webserver_interface=127.5.22.1
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:40263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.22.1
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:35:52.819965 13999 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:35:52.821717 13999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400: Bootstrap starting.
I20260928 06:35:52.838729 13999 log.cc:824] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:52.845388 13999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:35:52.845795 13999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400: Bootstrap complete.
I20260928 06:35:52.850064 13999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a40a2f78dd34f83b2dc371f81403400" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43529 } }
I20260928 06:35:52.850582 13999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a40a2f78dd34f83b2dc371f81403400, State: Initialized, Role: FOLLOWER
I20260928 06:35:52.851073 13999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a40a2f78dd34f83b2dc371f81403400" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43529 } }
I20260928 06:35:52.851186 13999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:52.851276 13999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:52.851377 13999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:35:52.853313 13999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a40a2f78dd34f83b2dc371f81403400" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43529 } }
I20260928 06:35:52.853658 13999 leader_election.cc:302] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a40a2f78dd34f83b2dc371f81403400; no voters:
I20260928 06:35:52.854008 13999 leader_election.cc:288] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:35:52.854161 14003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:35:52.854431 14003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [term 2 LEADER]: Becoming Leader. State: Replica: 8a40a2f78dd34f83b2dc371f81403400, State: Running, Role: LEADER
I20260928 06:35:52.854749 14003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a40a2f78dd34f83b2dc371f81403400" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43529 } }
I20260928 06:35:52.855276 13999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:52.856184 14004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8a40a2f78dd34f83b2dc371f81403400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a40a2f78dd34f83b2dc371f81403400" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43529 } } }
I20260928 06:35:52.856323 14004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:52.856683 14005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a40a2f78dd34f83b2dc371f81403400. Latest consensus state: current_term: 2 leader_uuid: "8a40a2f78dd34f83b2dc371f81403400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a40a2f78dd34f83b2dc371f81403400" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43529 } } }
I20260928 06:35:52.856823 14005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:52.857149 14012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:52.860188 14012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:52.860984 14012 catalog_manager.cc:1259] Loaded cluster ID: 12a8c65d49b8423296e58fa9acd632e0
I20260928 06:35:52.861047 14012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:52.863415 14012 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:52.864068 14012 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8a40a2f78dd34f83b2dc371f81403400: Loaded TSK: 0
I20260928 06:35:52.865250 14012 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:35:52.931872 14001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:52.932190 14001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:52.932265 14001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:52.936484 14001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:52.936585 14001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:52.936682 14001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:35:52.942165 14001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:40263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.1.keytab
--principal=oryx/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:36195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=34845
--tserver_master_addrs=127.5.22.62:43529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:52.943298 14001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:52.944347 14001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:52.955422 14026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:52.955438 14025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:52.955622 14028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:52.955778 14001 server_base.cc:1034] running on GCE node
I20260928 06:35:52.956205 14001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:52.957434 14001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:52.958658 14001 hybrid_clock.cc:648] HybridClock initialized: now 1790577352958635 us; error 49 us; skew 500 ppm
Sep 28 06:35:52 dist-test-slave-j0qf krb5kdc[13418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577352, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:52.961895 14001 init.cc:377] Logged in from keytab as oryx/127.5.22.1@KRBTEST.COM (short username oryx)
I20260928 06:35:52.963004 14001 webserver.cc:460] Webserver started at http://127.5.22.1:34845/ using document root <none> and password file <none>
I20260928 06:35:52.963634 14001 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:52.963763 14001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:52.968763 14001 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:35:52.971985 14035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:52.974078 14001 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 06:35:52.974224 14001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "3f3a026dea584e3a87cf81ba3a47643e"
format_stamp: "Formatted at 2026-09-28 06:35:52 on dist-test-slave-j0qf"
server_key: "2663ddabe4c555bebda60ffa8f789c30"
server_key_iv: "9cd69744f620a904dedcd7c8dd42de42"
server_key_version: "encryptionkey@0"
I20260928 06:35:52.975129 14001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:53.002903 14001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:53.006718 14001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:53.006889 14001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:53.007486 14001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:53.008360 14001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:53.008414 14001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:53.008466 14001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:53.008483 14001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:53.018070 14001 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:36195
I20260928 06:35:53.018095 14148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:36195 every 8 connection(s)
I20260928 06:35:53.019264 14001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:53.023722 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 14001
I20260928 06:35:53.024242 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:42063
--local_ip_for_outbound_sockets=127.5.22.2
--tserver_master_addrs=127.5.22.62:43529
--webserver_port=45655
--webserver_interface=127.5.22.2
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:40263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.22.2
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:35:53 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:53 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:35:53.035292 14151 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:53.021697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:38497 (local address 127.5.22.62:43529)
0928 06:35:53.022385 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:53.022405 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:53.022780 (+ 375us) server_negotiation.cc:407] Connection header received
0928 06:35:53.024027 (+ 1247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:53.024039 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:53.024390 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:53.024717 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:53.026383 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.028019 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.029376 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.030240 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.030468 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:53.034548 (+ 4080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.1:38497: BlockingRecv error: recv got EOF from 127.5.22.1:38497 (error 108)
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":182,"threads_started":1}
I20260928 06:35:53.035347 14152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:53.021801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.1:38497)
0928 06:35:53.022587 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:53.022635 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:53.023536 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:53.024883 (+ 1347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:53.024897 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:53.025480 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:53.026159 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:53.026176 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.028178 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.028181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:53.029231 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:53.029242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.030453 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.030457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:53.030596 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:53.031276 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:53.031294 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:53.034183 (+ 2889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":140,"threads_started":1}
W20260928 06:35:53.036029 14149 heartbeater.cc:644] Failed to heartbeat to 127.5.22.62:43529 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.22.62:43529: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Sep 28 06:35:53 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:53 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:35:53.042299 14152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:53.036534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.1:56503)
0928 06:35:53.036699 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:53.036713 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:53.036789 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:53.037010 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:53.037014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:53.037215 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:53.037327 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:53.037333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.038149 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.038152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:53.039054 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:53.039064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.039635 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.039639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:53.039732 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:53.040212 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:53.040227 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:53.042157 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101}
W20260928 06:35:53.042452 14151 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:53.036660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:56503 (local address 127.5.22.62:43529)
0928 06:35:53.036789 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:53.036792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:53.036806 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:35:53.036864 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:53.036866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:53.036907 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:53.036973 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:53.037486 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.038036 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.039199 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.039528 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.039615 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:53.042278 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.1:56503: BlockingRecv error: recv got EOF from 127.5.22.1:56503 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 06:35:53 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:53 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:35:53.047853 14152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:53.043056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.1:50435)
0928 06:35:53.043152 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:53.043168 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:53.043240 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:53.043462 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:53.043465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:53.043612 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:53.043675 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:53.043680 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.044391 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.044394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:53.044920 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:53.044928 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.045359 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.045363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:53.045455 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:53.045920 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:53.045932 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:53.047720 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 06:35:53.047930 14151 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:53.043146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:50435 (local address 127.5.22.62:43529)
0928 06:35:53.043253 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:53.043258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:53.043275 (+ 17us) server_negotiation.cc:407] Connection header received
0928 06:35:53.043313 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:53.043316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:53.043362 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:53.043447 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:53.043785 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.044277 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.045044 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.045253 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.045359 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:53.047828 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.1:50435: BlockingRecv error: recv got EOF from 127.5.22.1:50435 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260928 06:35:53.048051 14149 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:35:53.141373 14153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:53.141641 14153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:53.141685 14153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:53.145777 14153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:53.145849 14153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:53.145910 14153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:35:53.151428 14153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:40263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.2.keytab
--principal=oryx/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:42063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=45655
--tserver_master_addrs=127.5.22.62:43529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:53.152568 14153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:53.153666 14153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:53.164019 14161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:53.164050 14159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:53.164069 14158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:53.164345 14153 server_base.cc:1034] running on GCE node
I20260928 06:35:53.164824 14153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:53.165400 14153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:53.166584 14153 hybrid_clock.cc:648] HybridClock initialized: now 1790577353166547 us; error 47 us; skew 500 ppm
Sep 28 06:35:53 dist-test-slave-j0qf krb5kdc[13418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577353, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:53.169268 14153 init.cc:377] Logged in from keytab as oryx/127.5.22.2@KRBTEST.COM (short username oryx)
I20260928 06:35:53.170161 14153 webserver.cc:460] Webserver started at http://127.5.22.2:45655/ using document root <none> and password file <none>
I20260928 06:35:53.170783 14153 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:53.170861 14153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:53.174731 14153 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:35:53.176801 14168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:53.178053 14153 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:53.178150 14153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "da713a92dad743729f632197bbece7ca"
format_stamp: "Formatted at 2026-09-28 06:35:52 on dist-test-slave-j0qf"
server_key: "bc7b6d859ba4323e1088f63b1c1b4517"
server_key_iv: "1806e99835b2a3029b06d647caf2de3f"
server_key_version: "encryptionkey@0"
I20260928 06:35:53.178791 14153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:53.190678 14153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:53.193661 14153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:53.193813 14153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:53.194355 14153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:53.195303 14153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:53.195353 14153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:53.195395 14153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:53.195410 14153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:53.204495 14153 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:42063
I20260928 06:35:53.204555 14281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:42063 every 8 connection(s)
I20260928 06:35:53.205564 14153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:53.212009 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 14153
I20260928 06:35:53.212500 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:40555
--local_ip_for_outbound_sockets=127.5.22.3
--tserver_master_addrs=127.5.22.62:43529
--webserver_port=38129
--webserver_interface=127.5.22.3
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:40263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.22.3
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:35:53 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:53 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:35:53.217182 14151 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:53.207524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:55685 (local address 127.5.22.62:43529)
0928 06:35:53.207675 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:53.207679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:53.208852 (+ 1173us) server_negotiation.cc:407] Connection header received
0928 06:35:53.209750 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:53.209753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:53.209824 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:53.209928 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:53.211151 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.211697 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.213052 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.213278 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.213394 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:53.217054 (+ 3660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.2:55685: BlockingRecv error: recv got EOF from 127.5.22.2:55685 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 06:35:53.217845 14284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:53.208015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.2:55685)
0928 06:35:53.208679 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:53.208722 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:53.209533 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:53.210127 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:53.210134 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:53.210513 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:53.210991 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:53.211004 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.211830 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.211834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:53.212917 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:53.212926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.213434 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.213438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:53.213616 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:53.214242 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:53.214258 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:53.216911 (+ 2653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":119,"threads_started":1}
W20260928 06:35:53.218401 14282 heartbeater.cc:644] Failed to heartbeat to 127.5.22.62:43529 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.22.62:43529: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Sep 28 06:35:53 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:53 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:35:53.225308 14284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:53.218897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.2:42937)
0928 06:35:53.219104 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:53.219120 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:53.219209 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:53.219488 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:53.219494 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:53.219690 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:53.219857 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:53.219870 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.220632 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.220638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:53.221675 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:53.221689 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.222183 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.222190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:53.222349 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:53.222855 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:53.222876 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:53.225164 (+ 2288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120}
W20260928 06:35:53.225411 14151 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:53.219103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:42937 (local address 127.5.22.62:43529)
0928 06:35:53.219220 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:53.219223 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:53.219236 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:35:53.219324 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:53.219326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:53.219364 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:53.219433 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:53.219999 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.220512 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.221832 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.222060 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.222142 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:53.225282 (+ 3140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.2:42937: BlockingRecv error: recv got EOF from 127.5.22.2:42937 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 06:35:53 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:53 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:35:53.231283 14284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:53.226067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.2:34507)
0928 06:35:53.226175 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:53.226191 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:53.226262 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:53.226493 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:53.226496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:53.226618 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:53.226686 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:53.226690 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.227727 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.227730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:53.228248 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:53.228256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.228799 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.228802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:53.228857 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:53.229339 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:53.229350 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:53.231161 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 06:35:53.231480 14282 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:35:53.231534 14151 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:53.226159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:34507 (local address 127.5.22.62:43529)
0928 06:35:53.226272 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:53.226276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:53.226290 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:35:53.226352 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:53.226355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:53.226399 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:53.226480 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:53.226790 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.227619 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.228355 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.228696 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.228794 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:53.231298 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.2:34507: BlockingRecv error: recv got EOF from 127.5.22.2:34507 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 06:35:53.329749 14285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:53.330017 14285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:53.330061 14285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:53.334053 14285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:53.334116 14285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:53.334174 14285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:35:53.339449 14285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:40263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/krb5kdc/oryx_127.5.22.3.keytab
--principal=oryx/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:40555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=38129
--tserver_master_addrs=127.5.22.62:43529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:35:53.340562 14285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:53.341517 14285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:53.351207 14290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:53.351203 14291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:53.351203 14293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:53.351613 14285 server_base.cc:1034] running on GCE node
I20260928 06:35:53.352051 14285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:53.352736 14285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:53.353926 14285 hybrid_clock.cc:648] HybridClock initialized: now 1790577353353887 us; error 52 us; skew 500 ppm
Sep 28 06:35:53 dist-test-slave-j0qf krb5kdc[13418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577353, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:53.357087 14285 init.cc:377] Logged in from keytab as oryx/127.5.22.3@KRBTEST.COM (short username oryx)
I20260928 06:35:53.358018 14285 webserver.cc:460] Webserver started at http://127.5.22.3:38129/ using document root <none> and password file <none>
I20260928 06:35:53.358671 14285 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:53.358773 14285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:53.362818 14285 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260928 06:35:53.364741 14300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:53.365938 14285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:53.366077 14285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "4cd1cf0ab5904e15966e38903de50c7e"
format_stamp: "Formatted at 2026-09-28 06:35:52 on dist-test-slave-j0qf"
server_key: "b0ff9017c98506701f33e95ca4608045"
server_key_iv: "2ca5000fc7bddd934c24eee63bd5a488"
server_key_version: "encryptionkey@0"
I20260928 06:35:53.366703 14285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:53.392700 14285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:53.395586 14285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:53.395742 14285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:53.396323 14285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:53.397146 14285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:53.397193 14285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:53.397233 14285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:53.397248 14285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:53.405821 14285 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:40555
I20260928 06:35:53.405893 14413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:40555 every 8 connection(s)
I20260928 06:35:53.406894 14285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:53.409952 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 14285
Sep 28 06:35:53 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:53 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:35:53.417984 14151 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:53.408844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:59079 (local address 127.5.22.62:43529)
0928 06:35:53.408984 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:53.408988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:53.410112 (+ 1124us) server_negotiation.cc:407] Connection header received
0928 06:35:53.411254 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:53.411258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:53.411315 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:53.411418 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:53.412586 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.413098 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.414227 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.414513 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.414652 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:53.417862 (+ 3210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.3:59079: BlockingRecv error: recv got EOF from 127.5.22.3:59079 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:35:53.418761 14416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:53.409257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.3:59079)
0928 06:35:53.409913 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:53.409954 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:53.411035 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:53.411613 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:53.411620 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:53.411958 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:53.412439 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:53.412451 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.413221 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.413225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:53.414100 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:53.414109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.414689 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.414694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:53.414866 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:53.415418 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:53.415433 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:53.417762 (+ 2329us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":114,"threads_started":1}
W20260928 06:35:53.419318 14414 heartbeater.cc:644] Failed to heartbeat to 127.5.22.62:43529 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.22.62:43529: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
I20260928 06:35:53.420117 14418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:53.411020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39820 (local address 127.5.22.62:43529)
0928 06:35:53.411293 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:53.411297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:53.411308 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:35:53.411342 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:53.411344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:53.411377 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:53.411476 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:53.412219 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.412783 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.413672 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.413915 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.414028 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:53.415668 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:53.415674 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:53.415688 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:53.415714 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:53.417598 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:53.418049 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:53.418056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:53.418062 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:53.418290 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:53.418621 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:53.418625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:53.418626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:53.419086 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:53.419286 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:53.419634 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 06:35:53.419915 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":134,"threads_started":1}
Sep 28 06:35:53 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:53 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:35:53.425897 14416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:53.419796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.3:53829)
0928 06:35:53.419914 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:53.419930 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:53.420004 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:53.420801 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:53.420805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:53.420963 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:53.421070 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:53.421077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.422094 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.422097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:53.422883 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:53.422892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.423444 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.423447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:53.423518 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:53.424028 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:53.424049 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:53.425798 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:35:53.426095 14418 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:53.420251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:53829 (local address 127.5.22.62:43529)
0928 06:35:53.420517 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:53.420521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:53.420533 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:35:53.420566 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:53.420569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:53.420618 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:53.420693 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:53.421195 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.421955 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.422992 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.423328 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.423420 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:53.425959 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.3:53829: BlockingRecv error: recv got EOF from 127.5.22.3:53829 (error 108)
Metrics: {"server-negotiator.queue_time_us":188}
Sep 28 06:35:53 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:53 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:35:53.431124 14416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:53.426545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.3:58067)
0928 06:35:53.426671 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:53.426687 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:53.426762 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:53.426972 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:53.426974 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:53.427100 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:53.427171 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:53.427175 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.427869 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.427871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:53.428373 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:53.428379 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.428786 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.428788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:53.428843 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:53.429278 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:53.429288 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:53.431014 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 06:35:53.431255 14418 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:53.426624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:58067 (local address 127.5.22.62:43529)
0928 06:35:53.426781 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:53.426784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:53.426817 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:35:53.426848 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:53.426850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:53.426883 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:53.426938 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:53.427293 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.427779 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.428477 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.428689 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.428769 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:53.431148 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.3:58067: BlockingRecv error: recv got EOF from 127.5.22.3:58067 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260928 06:35:53.431468 14414 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 06:35:54 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:54 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:35:54.054600 14419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.048689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.1:47065)
0928 06:35:54.049006 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.049020 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.049118 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.049372 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.049374 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.049553 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:54.049699 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.049704 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.050523 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.050526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.051110 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.051119 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.051639 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.051641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.051716 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.052277 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:54.052289 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:54.054404 (+ 2115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":88,"threads_started":1}
W20260928 06:35:54.054659 14420 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:54.048810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:47065 (local address 127.5.22.62:43529)
0928 06:35:54.049064 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.049068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.049083 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:35:54.049211 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.049213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.049258 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.049336 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.049806 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.050427 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.051247 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.051513 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.051608 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.054530 (+ 2922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.1:47065: BlockingRecv error: recv got EOF from 127.5.22.1:47065 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":60,"threads_started":1}
Sep 28 06:35:54 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:54 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:35:54.237815 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.232085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.2:49117)
0928 06:35:54.232336 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.232365 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.232454 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.232756 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.232758 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.232928 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:54.233022 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.233027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.233752 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.233754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.234441 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.234451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.234903 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.234905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.234966 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.235466 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:54.235478 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:54.237641 (+ 2163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":68,"threads_started":1}
W20260928 06:35:54.238001 14420 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:54.232186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:49117 (local address 127.5.22.62:43529)
0928 06:35:54.232395 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.232399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.232414 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:35:54.232580 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.232582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.232629 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.232700 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.233125 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.233658 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.234596 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.234806 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.234884 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.237829 (+ 2945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.2:49117: BlockingRecv error: recv got EOF from 127.5.22.2:49117 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 28 06:35:54 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:54 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:35:54.442075 14422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.432102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.3:54467)
0928 06:35:54.432448 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.432462 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.432602 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.432890 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.432893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.433114 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:54.433229 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.433235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.434024 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.434030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.434644 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.434652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.435134 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.435138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.435231 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.439839 (+ 4608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:54.439851 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:54.441896 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":92,"threads_started":1}
W20260928 06:35:54.442157 14420 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:54.432242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:54467 (local address 127.5.22.62:43529)
0928 06:35:54.432412 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.432417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.432547 (+ 130us) server_negotiation.cc:407] Connection header received
0928 06:35:54.432713 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.432718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.432768 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.432888 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.433351 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.433908 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.434769 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.435020 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.435130 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.442040 (+ 6910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.3:54467: BlockingRecv error: recv got EOF from 127.5.22.3:54467 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 06:35:55 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:55 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:35:55.061308 14423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:55.055425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.1:59077)
0928 06:35:55.055724 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:55.055739 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:55.055847 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:55.056135 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:55.056137 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:55.056367 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:55.056506 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:55.056517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.057511 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.057514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:55.058145 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:55.058155 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.058745 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.058748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:55.058810 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:55.059381 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:55.059393 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:55.061172 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":104,"threads_started":1}
W20260928 06:35:55.061532 14424 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:55.055592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:59077 (local address 127.5.22.62:43529)
0928 06:35:55.055888 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:55.055893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:55.055916 (+ 23us) server_negotiation.cc:407] Connection header received
0928 06:35:55.055954 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:55.055956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:55.056001 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:55.056089 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:55.056668 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.057346 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.058270 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.058633 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.058721 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:55.061329 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.1:59077: BlockingRecv error: recv got EOF from 127.5.22.1:59077 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":90,"threads_started":1}
Sep 28 06:35:55 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:55 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:35:55.244844 14425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:55.238738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.2:50303)
0928 06:35:55.239021 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:55.239043 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:55.239175 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:55.239463 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:55.239466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:55.239691 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:55.239786 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:55.239791 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.240609 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.240614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:55.241196 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:55.241204 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.241669 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.241672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:55.241762 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:55.242379 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:55.242391 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:55.244642 (+ 2251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":104,"threads_started":1}
W20260928 06:35:55.244870 14424 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:55.238805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:50303 (local address 127.5.22.62:43529)
0928 06:35:55.238962 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:55.238969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:55.239136 (+ 167us) server_negotiation.cc:407] Connection header received
0928 06:35:55.239293 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:55.239296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:55.239351 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:55.239443 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:55.239908 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.240486 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.241321 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.241553 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.241666 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:55.244769 (+ 3103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.2:50303: BlockingRecv error: recv got EOF from 127.5.22.2:50303 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 06:35:55 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:55 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:35:55.453266 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:55.442854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.3:60581)
0928 06:35:55.443124 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:55.443138 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:55.443226 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:55.443447 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:55.443449 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:55.443639 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:55.443736 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:55.443742 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.444457 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.444459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:55.445013 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:55.445020 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.445424 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.445426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:55.445483 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:55.451213 (+ 5730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:55.451232 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:55.453126 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":94,"threads_started":1}
W20260928 06:35:55.453435 14424 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:55.443045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:60581 (local address 127.5.22.62:43529)
0928 06:35:55.443183 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:55.443187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:55.443200 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:35:55.443293 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:55.443295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:55.443338 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:55.443407 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:55.443837 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.444361 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.445117 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.445327 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.445403 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:55.453274 (+ 7871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.3:60581: BlockingRecv error: recv got EOF from 127.5.22.3:60581 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 06:35:56 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:56 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:35:56.067400 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:56.062072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.1:54427)
0928 06:35:56.062417 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:56.062432 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:56.062534 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:56.062777 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:56.062779 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:56.062969 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:56.063063 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:56.063068 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:56.063797 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:56.063799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:56.064373 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:56.064381 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:56.064798 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:56.064800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:56.064861 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:56.065351 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:56.065362 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:56.067288 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":88,"threads_started":1}
W20260928 06:35:56.067531 14428 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:56.062151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:54427 (local address 127.5.22.62:43529)
0928 06:35:56.062446 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:56.062450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:56.062504 (+ 54us) server_negotiation.cc:407] Connection header received
0928 06:35:56.062618 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:56.062620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:56.062664 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:56.062739 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:56.063160 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:56.063703 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:56.064474 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:56.064700 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:56.064775 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:56.067422 (+ 2647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.1:54427: BlockingRecv error: recv got EOF from 127.5.22.1:54427 (error 108)
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":107,"threads_started":1}
Sep 28 06:35:56 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:56 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:35:56.250819 14429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:56.245663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.2:38221)
0928 06:35:56.245939 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:56.245952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:56.246048 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:56.246297 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:56.246299 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:56.246522 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:56.246624 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:56.246630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:56.247356 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:56.247358 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:56.247906 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:56.247913 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:56.248316 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:56.248319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:56.248379 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:56.248859 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:56.248870 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:56.250679 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":72,"threads_started":1}
W20260928 06:35:56.250914 14428 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:56.245809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:38221 (local address 127.5.22.62:43529)
0928 06:35:56.245966 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:56.245970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:56.246036 (+ 66us) server_negotiation.cc:407] Connection header received
0928 06:35:56.246143 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:56.246145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:56.246191 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:56.246258 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:56.246732 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:56.247263 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:56.248007 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:56.248221 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:56.248296 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:56.250822 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.2:38221: BlockingRecv error: recv got EOF from 127.5.22.2:38221 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 06:35:56 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:56 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:35:56.459651 14430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:56.454009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.3:49529)
0928 06:35:56.454330 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:56.454344 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:56.454449 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:56.454718 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:56.454721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:56.454964 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:56.455067 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:56.455073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:56.455918 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:56.455942 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:56.456566 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:56.456575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:56.457140 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:56.457143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:56.457240 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:56.457751 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:56.457762 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:56.459532 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":80,"threads_started":1}
W20260928 06:35:56.459826 14428 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:56.454053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:49529 (local address 127.5.22.62:43529)
0928 06:35:56.454184 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:56.454189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:56.454423 (+ 234us) server_negotiation.cc:407] Connection header received
0928 06:35:56.454517 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:56.454519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:56.454574 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:56.454651 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:56.455192 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:56.455814 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:56.456696 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:56.456996 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:56.457146 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:56.459672 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.3:49529: BlockingRecv error: recv got EOF from 127.5.22.3:49529 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 06:35:57 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:57 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:35:57.074380 14432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:57.068288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.1:59005)
0928 06:35:57.068653 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:57.068667 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:57.068768 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:57.069026 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:57.069028 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:57.069275 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:57.069373 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:57.069379 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:57.070196 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:57.070199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:57.070848 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:57.070857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:57.071357 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:57.071360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:57.071422 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:57.071976 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:57.071987 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:57.074217 (+ 2230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":163,"threads_started":1}
W20260928 06:35:57.074576 14431 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:57.068359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:59005 (local address 127.5.22.62:43529)
0928 06:35:57.068664 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:57.068668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:57.068720 (+ 52us) server_negotiation.cc:407] Connection header received
0928 06:35:57.068851 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:57.068853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:57.068913 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:57.068991 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:57.069498 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:57.070045 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:57.070962 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:57.071186 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:57.071342 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:57.074419 (+ 3077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.1:59005: BlockingRecv error: recv got EOF from 127.5.22.1:59005 (error 108)
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":102,"threads_started":1}
Sep 28 06:35:57 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:57 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:35:57.257385 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:57.251725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.2:41757)
0928 06:35:57.251997 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:57.252011 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:57.252128 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:57.252373 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:57.252375 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:57.252550 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:57.252652 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:57.252658 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:57.253442 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:57.253444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:57.254001 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:57.254010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:57.254454 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:57.254456 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:57.254519 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:57.255066 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:57.255079 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:57.257220 (+ 2141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":102,"threads_started":1}
W20260928 06:35:57.257501 14431 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:57.251839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:41757 (local address 127.5.22.62:43529)
0928 06:35:57.251995 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:57.251999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:57.252065 (+ 66us) server_negotiation.cc:407] Connection header received
0928 06:35:57.252206 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:57.252209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:57.252252 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:57.252330 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:57.252794 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:57.253339 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:57.254113 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:57.254353 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:57.254434 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:57.257383 (+ 2949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.2:41757: BlockingRecv error: recv got EOF from 127.5.22.2:41757 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 06:35:57 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:57 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:35:57.466542 14434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:57.460451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.3:51207)
0928 06:35:57.460790 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:57.460811 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:57.460946 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:57.461227 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:57.461230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:57.461441 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:57.461532 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:57.461538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:57.462311 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:57.462346 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:57.462897 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:57.462906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:57.463407 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:57.463410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:57.463493 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:57.464123 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:57.464141 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:57.466417 (+ 2276us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":117,"threads_started":1}
W20260928 06:35:57.466684 14431 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:57.460574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:51207 (local address 127.5.22.62:43529)
0928 06:35:57.460746 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:57.460751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:57.460886 (+ 135us) server_negotiation.cc:407] Connection header received
0928 06:35:57.461049 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:57.461053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:57.461112 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:57.461202 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:57.461677 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:57.462200 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:57.463050 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:57.463285 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:57.463395 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:57.466551 (+ 3156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.3:51207: BlockingRecv error: recv got EOF from 127.5.22.3:51207 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 06:35:58 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:58 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:35:58.081094 14435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.075305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.1:39305)
0928 06:35:58.075728 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:58.075746 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:58.075867 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:58.076233 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:58.076237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:58.076484 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:58.076606 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:58.076621 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.077360 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.077362 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:58.077928 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:58.077935 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.078419 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.078425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:58.078517 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.079083 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:58.079104 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:58.080957 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":91,"threads_started":1}
W20260928 06:35:58.081280 14436 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:58.075388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:39305 (local address 127.5.22.62:43529)
0928 06:35:58.075682 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:58.075688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:58.075834 (+ 146us) server_negotiation.cc:407] Connection header received
0928 06:35:58.075986 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:58.075989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:58.076074 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:58.076170 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:58.076728 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.077269 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.078043 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.078262 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.078411 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.081105 (+ 2694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.1:39305: BlockingRecv error: recv got EOF from 127.5.22.1:39305 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":92,"threads_started":1}
Sep 28 06:35:58 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:58 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:35:58.263975 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.258234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.2:48235)
0928 06:35:58.258567 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:58.258589 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:58.258680 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:58.258919 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:58.258921 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:58.259109 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:58.259203 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:58.259209 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.259946 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.259949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:58.260529 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:58.260538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.260966 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.260969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:58.261046 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.261622 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:58.261634 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:58.263804 (+ 2170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":167,"threads_started":1}
W20260928 06:35:58.264029 14436 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:58.258388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:48235 (local address 127.5.22.62:43529)
0928 06:35:58.258535 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:58.258539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:58.258657 (+ 118us) server_negotiation.cc:407] Connection header received
0928 06:35:58.258763 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:58.258765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:58.258809 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:58.258882 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:58.259304 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.259842 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.260656 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.260870 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.260951 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.263924 (+ 2973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.2:48235: BlockingRecv error: recv got EOF from 127.5.22.2:48235 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 06:35:58 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:58 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:35:58.472769 14438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.467375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.3:41525)
0928 06:35:58.467666 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:58.467678 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:58.467770 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:58.468023 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:58.468026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:58.468202 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:58.468295 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:58.468301 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.469040 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.469043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:58.469615 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:58.469629 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.470163 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.470165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:58.470225 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.470779 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:58.470791 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:58.472641 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":116,"threads_started":1}
W20260928 06:35:58.472898 14436 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:58.467473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:41525 (local address 127.5.22.62:43529)
0928 06:35:58.467620 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:58.467624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:58.467743 (+ 119us) server_negotiation.cc:407] Connection header received
0928 06:35:58.467867 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:58.467869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:58.467916 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:58.467986 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:58.468395 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.468937 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.469775 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.470057 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.470146 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.472779 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.3:41525: BlockingRecv error: recv got EOF from 127.5.22.3:41525 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 06:35:59 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:59 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:35:59.087670 14439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.081971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.1:52463)
0928 06:35:59.082401 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:59.082417 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:59.082560 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:59.082927 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:59.082930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:59.083223 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:59.083354 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.083360 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.084110 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.084112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.084669 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.084677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.085091 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.085093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.085151 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.085673 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:59.085685 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:59.087555 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":131,"threads_started":1}
W20260928 06:35:59.087832 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:59.082039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:52463 (local address 127.5.22.62:43529)
0928 06:35:59.082451 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:59.082455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:59.082493 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:35:59.082707 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:59.082711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:59.082772 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:59.082866 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:59.083467 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.084011 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.084776 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.084995 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.085071 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.087690 (+ 2619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.1:52463: BlockingRecv error: recv got EOF from 127.5.22.1:52463 (error 108)
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":111,"threads_started":1}
Sep 28 06:35:59 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:59 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:35:59.270850 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.264933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.2:37227)
0928 06:35:59.265207 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:59.265221 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:59.265364 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:59.265649 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:59.265652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:59.265858 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:59.265948 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.265954 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.266799 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.266804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.267365 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.267372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.267843 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.267847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.267941 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.268537 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:59.268552 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:59.270734 (+ 2182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":76,"threads_started":1}
W20260928 06:35:59.271042 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:59.264969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:37227 (local address 127.5.22.62:43529)
0928 06:35:59.265152 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:59.265157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:59.265300 (+ 143us) server_negotiation.cc:407] Connection header received
0928 06:35:59.265476 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:59.265484 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:59.265531 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:59.265624 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:59.266071 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.266688 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.267490 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.267722 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.267829 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.270913 (+ 3084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.2:37227: BlockingRecv error: recv got EOF from 127.5.22.2:37227 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 06:35:59 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:35:59 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:35:59.478937 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.473506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.3:47225)
0928 06:35:59.473801 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:59.473815 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:59.473914 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:59.474163 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:59.474165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:59.474409 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:59.474512 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.474518 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.475271 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.475275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.475832 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.475840 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.476283 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.476286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.476348 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.476873 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:59.476885 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:59.478811 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":122,"threads_started":1}
W20260928 06:35:59.479084 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:59.473615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:47225 (local address 127.5.22.62:43529)
0928 06:35:59.473753 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:59.473757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:59.473886 (+ 129us) server_negotiation.cc:407] Connection header received
0928 06:35:59.473994 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:59.473996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:59.474047 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:59.474108 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:59.474624 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.475149 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.475954 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.476175 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.476255 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.478947 (+ 2692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.3:47225: BlockingRecv error: recv got EOF from 127.5.22.3:47225 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 06:36:00 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:00 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:00.094244 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.088487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.1:47937)
0928 06:36:00.088842 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.088857 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.088950 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.089208 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.089211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.089413 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:00.089509 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.089515 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.090442 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.090446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.091060 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.091071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.091573 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.091577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.091665 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.092245 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:00.092257 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:00.094073 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":97,"threads_started":1}
W20260928 06:36:00.094343 14444 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:00.088681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:47937 (local address 127.5.22.62:43529)
0928 06:36:00.088941 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.088945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.088961 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:36:00.089026 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.089029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.089074 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.089178 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.089649 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.090264 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.091202 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.091449 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.091569 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.094196 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.1:47937: BlockingRecv error: recv got EOF from 127.5.22.1:47937 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":92,"threads_started":1}
Sep 28 06:36:00 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:00 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:00.277185 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.271779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.2:37085)
0928 06:36:00.272091 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.272110 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.272206 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.272510 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.272512 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.272727 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:00.272822 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.272829 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.273614 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.273616 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.274168 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.274176 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.274604 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.274607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.274669 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.275148 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:00.275159 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:00.277066 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":91,"threads_started":1}
W20260928 06:36:00.277369 14444 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:00.271894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:37085 (local address 127.5.22.62:43529)
0928 06:36:00.272057 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.272061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.272181 (+ 120us) server_negotiation.cc:407] Connection header received
0928 06:36:00.272271 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.272273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.272320 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.272487 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.272950 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.273498 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.274272 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.274509 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.274610 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.277209 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.2:37085: BlockingRecv error: recv got EOF from 127.5.22.2:37085 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 06:36:00 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:00 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:00.488898 14446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.479771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.3:55741)
0928 06:36:00.480080 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.480093 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.480192 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.480475 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.480478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.480732 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:00.480870 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.480880 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.481967 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.481973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.482587 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.482596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.483119 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.483122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.483183 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.486883 (+ 3700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:00.486904 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:00.488780 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":92,"threads_started":1}
W20260928 06:36:00.489089 14444 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:00.479827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:55741 (local address 127.5.22.62:43529)
0928 06:36:00.480000 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.480009 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.480164 (+ 155us) server_negotiation.cc:407] Connection header received
0928 06:36:00.480284 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.480287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.480346 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.480422 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.481014 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.481816 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.482685 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.483002 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.483111 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.488925 (+ 5814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.3:55741: BlockingRecv error: recv got EOF from 127.5.22.3:55741 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 28 06:36:01 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:01 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:01.100808 14447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.095126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.1:56937)
0928 06:36:01.095427 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:01.095444 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:01.095550 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:01.095811 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:01.095813 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:01.096043 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:01.096155 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.096160 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.097043 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.097046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.097725 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.097736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.098164 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.098166 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.098226 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.098813 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:01.098825 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:01.100692 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":113,"threads_started":1}
W20260928 06:36:01.100992 14448 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:01.095248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:56937 (local address 127.5.22.62:43529)
0928 06:36:01.095487 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:01.095491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:01.095505 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:36:01.095639 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:01.095641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:01.095686 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:01.095765 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:01.096263 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.096892 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.097834 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.098061 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.098144 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.100849 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.1:56937: BlockingRecv error: recv got EOF from 127.5.22.1:56937 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":91,"threads_started":1}
Sep 28 06:36:01 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:01 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:01.284085 14449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.277974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.2:38745)
0928 06:36:01.278333 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:01.278347 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:01.278442 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:01.278691 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:01.278694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:01.278937 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:01.279037 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.279042 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.280227 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.280233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.280887 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.280896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.281377 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.281381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.281469 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.282008 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:01.282022 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:01.283901 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":123,"threads_started":1}
W20260928 06:36:01.284181 14448 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:01.278096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:38745 (local address 127.5.22.62:43529)
0928 06:36:01.278240 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:01.278244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:01.278412 (+ 168us) server_negotiation.cc:407] Connection header received
0928 06:36:01.278527 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:01.278530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:01.278581 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:01.278654 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:01.279143 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.280130 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.281027 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.281265 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.281374 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.284042 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.2:38745: BlockingRecv error: recv got EOF from 127.5.22.2:38745 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 06:36:01 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:01 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:01.495476 14450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.489878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.3:45103)
0928 06:36:01.490147 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:01.490165 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:01.490289 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:01.490641 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:01.490645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:01.490836 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:01.490934 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.490940 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.491767 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.491770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.492326 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.492345 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.492785 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.492788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.492854 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.493441 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:01.493453 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:01.495357 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":103,"threads_started":1}
W20260928 06:36:01.495707 14448 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:01.489937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:45103 (local address 127.5.22.62:43529)
0928 06:36:01.490080 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:01.490085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:01.490268 (+ 183us) server_negotiation.cc:407] Connection header received
0928 06:36:01.490408 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:01.490411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:01.490474 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:01.490577 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:01.491073 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.491663 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.492446 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.492679 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.492759 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.495558 (+ 2799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.3:45103: BlockingRecv error: recv got EOF from 127.5.22.3:45103 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 06:36:02 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:02 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:02.107697 14451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.101782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.1:46597)
0928 06:36:02.102074 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.102089 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:02.102180 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:02.102477 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:02.102480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:02.102667 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:02.102773 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:02.102778 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.103830 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.103834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:02.104444 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:02.104478 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.104916 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.104918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:02.104982 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:02.105450 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:02.105461 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:02.107492 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":94,"threads_started":1}
W20260928 06:36:02.107739 14452 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:02.101852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:46597 (local address 127.5.22.62:43529)
0928 06:36:02.102124 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:02.102128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:02.102142 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:36:02.102276 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:02.102278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:02.102354 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:02.102435 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:02.102880 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.103566 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.104595 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.104818 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.104898 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:02.107610 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.1:46597: BlockingRecv error: recv got EOF from 127.5.22.1:46597 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":116,"threads_started":1}
Sep 28 06:36:02 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:02 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:02.290793 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.284838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.2:33367)
0928 06:36:02.285153 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.285175 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:02.285304 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:02.285687 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:02.285690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:02.285913 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:02.286006 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:02.286011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.286834 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.286839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:02.287411 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:02.287419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.287893 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.287896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:02.287991 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:02.288522 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:02.288534 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:02.290618 (+ 2084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":100,"threads_started":1}
W20260928 06:36:02.290853 14452 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:02.284930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:33367 (local address 127.5.22.62:43529)
0928 06:36:02.285080 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:02.285085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:02.285249 (+ 164us) server_negotiation.cc:407] Connection header received
0928 06:36:02.285516 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:02.285519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:02.285575 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:02.285665 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:02.286123 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.286726 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.287542 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.287776 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.287895 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:02.290743 (+ 2848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.2:33367: BlockingRecv error: recv got EOF from 127.5.22.2:33367 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 06:36:02 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:02 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:02.501752 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.496246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.3:42655)
0928 06:36:02.496519 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.496537 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:02.496679 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:02.496936 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:02.496938 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:02.497113 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:02.497218 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:02.497224 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.498075 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.498078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:02.498658 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:02.498666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.499080 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.499083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:02.499143 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:02.499749 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:02.499761 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:02.501619 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":113,"threads_started":1}
W20260928 06:36:02.501904 14452 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:02.496354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:42655 (local address 127.5.22.62:43529)
0928 06:36:02.496498 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:02.496501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:02.496623 (+ 122us) server_negotiation.cc:407] Connection header received
0928 06:36:02.496765 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:02.496768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:02.496814 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:02.496883 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:02.497326 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.497972 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.498764 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.498981 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.499065 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:02.501786 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.3:42655: BlockingRecv error: recv got EOF from 127.5.22.3:42655 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 06:36:03 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:03 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:03.114075 14455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:03.108535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.1:37479)
0928 06:36:03.108857 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:03.108872 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:03.108959 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:03.109310 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:03.109312 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:03.109490 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:03.109584 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:03.109589 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:03.110417 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:03.110419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:03.111045 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:03.111056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:03.111486 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:03.111489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:03.111550 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:03.112070 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:03.112081 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:03.113950 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":93,"threads_started":1}
W20260928 06:36:03.114391 14456 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:03.108687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:37479 (local address 127.5.22.62:43529)
0928 06:36:03.108933 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:03.108937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:03.108950 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:36:03.109051 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:03.109053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:03.109104 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:03.109214 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:03.109693 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:03.110245 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:03.111163 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:03.111386 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:03.111468 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:03.114109 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.1:37479: BlockingRecv error: recv got EOF from 127.5.22.1:37479 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":97,"threads_started":1}
Sep 28 06:36:03 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:03 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:03.297293 14457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:03.291896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.2:33001)
0928 06:36:03.292170 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:03.292184 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:03.292276 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:03.292648 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:03.292650 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:03.292833 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:03.292933 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:03.292939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:03.293656 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:03.293659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:03.294219 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:03.294227 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:03.294666 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:03.294669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:03.294735 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:03.295242 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:03.295253 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:03.297157 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":79,"threads_started":1}
W20260928 06:36:03.297443 14456 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:03.291950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:33001 (local address 127.5.22.62:43529)
0928 06:36:03.292139 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:03.292144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:03.292409 (+ 265us) server_negotiation.cc:407] Connection header received
0928 06:36:03.292455 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:03.292457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:03.292517 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:03.292601 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:03.293037 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:03.293559 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:03.294352 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:03.294568 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:03.294643 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:03.297324 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.2:33001: BlockingRecv error: recv got EOF from 127.5.22.2:33001 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 28 06:36:03 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:03 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:03.507841 14458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:03.502718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.3:57235)
0928 06:36:03.503007 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:03.503024 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:03.503140 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:03.503359 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:03.503361 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:03.503538 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:03.503632 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:03.503637 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:03.504361 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:03.504363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:03.504907 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:03.504914 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:03.505341 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:03.505343 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:03.505403 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:03.505884 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:03.505895 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:03.507728 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":117,"threads_started":1}
W20260928 06:36:03.507978 14456 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:03.502858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:57235 (local address 127.5.22.62:43529)
0928 06:36:03.503001 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:03.503005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:03.503095 (+ 90us) server_negotiation.cc:407] Connection header received
0928 06:36:03.503217 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:03.503219 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:03.503264 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:03.503324 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:03.503734 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:03.504258 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:03.505018 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:03.505241 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:03.505323 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:03.507861 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.3:57235: BlockingRecv error: recv got EOF from 127.5.22.3:57235 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 06:36:04 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:04 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:04.120913 14459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.114941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.1:47819)
0928 06:36:04.115354 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:04.115368 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:04.115465 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:04.115807 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:04.115810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:04.116022 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:04.116124 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.116130 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.116950 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.116953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.117575 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.117583 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.118023 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.118027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.118096 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.118733 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:04.118746 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:04.120787 (+ 2041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":120,"threads_started":1}
W20260928 06:36:04.121093 14460 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:04.115139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:47819 (local address 127.5.22.62:43529)
0928 06:36:04.115385 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:04.115389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:04.115420 (+ 31us) server_negotiation.cc:407] Connection header received
0928 06:36:04.115585 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:04.115587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:04.115631 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:04.115714 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:04.116277 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.116830 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.117686 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.117909 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.117992 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.120933 (+ 2941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.1:47819: BlockingRecv error: recv got EOF from 127.5.22.1:47819 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":63,"threads_started":1}
Sep 28 06:36:04 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:04 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:04.304386 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.298142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.2:54843)
0928 06:36:04.298585 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:04.298600 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:04.298739 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:04.299111 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:04.299115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:04.299343 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:04.299476 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.299482 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.300315 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.300322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.300947 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.300956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.301484 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.301489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.301593 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.302200 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:04.302213 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:04.304166 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":84,"threads_started":1}
W20260928 06:36:04.304400 14460 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:04.298271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:54843 (local address 127.5.22.62:43529)
0928 06:36:04.298459 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:04.298464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:04.298687 (+ 223us) server_negotiation.cc:407] Connection header received
0928 06:36:04.298890 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:04.298893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:04.298954 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:04.299086 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:04.299601 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.300197 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.301076 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.301354 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.301478 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.304294 (+ 2816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.2:54843: BlockingRecv error: recv got EOF from 127.5.22.2:54843 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 06:36:04 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:04 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:04.514143 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.508746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.3:43413)
0928 06:36:04.509021 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:04.509035 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:04.509134 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:04.509363 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:04.509365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:04.509537 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:04.509656 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.509662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.510426 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.510428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.511000 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.511009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.511492 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.511494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.511557 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.512061 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:04.512073 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:04.514003 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":90,"threads_started":1}
W20260928 06:36:04.514312 14460 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:04.508834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:43413 (local address 127.5.22.62:43529)
0928 06:36:04.509025 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:04.509029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:04.509087 (+ 58us) server_negotiation.cc:407] Connection header received
0928 06:36:04.509221 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:04.509223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:04.509270 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:04.509340 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:04.509760 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.510299 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.511141 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.511383 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.511478 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.514178 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.3:43413: BlockingRecv error: recv got EOF from 127.5.22.3:43413 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 28 06:36:05 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:05 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:05.127743 14463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:05.121945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.1:33017)
0928 06:36:05.122300 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:05.122349 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:05.122472 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:05.122831 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:05.122834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:05.123084 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:05.123230 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:05.123240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.124136 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.124139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:05.124713 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:05.124722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.125155 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.125158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:05.125221 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:05.125707 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:05.125719 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:05.127626 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":119,"threads_started":1}
W20260928 06:36:05.127946 14464 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:05.122018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:33017 (local address 127.5.22.62:43529)
0928 06:36:05.122300 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:05.122304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:05.122454 (+ 150us) server_negotiation.cc:407] Connection header received
0928 06:36:05.122580 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:05.122583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:05.122647 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:05.122749 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:05.123364 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.124006 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.124825 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.125051 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.125131 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:05.127766 (+ 2635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.1:33017: BlockingRecv error: recv got EOF from 127.5.22.1:33017 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":64,"threads_started":1}
Sep 28 06:36:05 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:05 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:05.311256 14465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:05.305268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.2:60643)
0928 06:36:05.305612 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:05.305625 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:05.305755 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:05.306116 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:05.306119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:05.306411 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:05.306523 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:05.306530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.307293 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.307298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:05.307877 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:05.307885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.308351 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.308354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:05.308440 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:05.308941 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:05.308952 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:05.311080 (+ 2128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":88,"threads_started":1}
W20260928 06:36:05.311308 14464 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:05.305344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:60643 (local address 127.5.22.62:43529)
0928 06:36:05.305545 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:05.305550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:05.305701 (+ 151us) server_negotiation.cc:407] Connection header received
0928 06:36:05.305861 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:05.305866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:05.305919 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:05.306092 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:05.306637 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.307180 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.308004 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.308238 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.308348 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:05.311203 (+ 2855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.2:60643: BlockingRecv error: recv got EOF from 127.5.22.2:60643 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 28 06:36:05 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:05 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:05.524667 14466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:05.515008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.3:43267)
0928 06:36:05.515288 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:05.515302 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:05.515478 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:05.515754 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:05.515757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:05.515951 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:05.516057 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:05.516063 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.516834 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.516837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:05.517518 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:05.517528 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.517959 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.517962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:05.518028 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:05.522575 (+ 4547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:05.522587 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:05.524538 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":99,"threads_started":1}
W20260928 06:36:05.524832 14464 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:05.515138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:43267 (local address 127.5.22.62:43529)
0928 06:36:05.515274 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:05.515277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:05.515371 (+ 94us) server_negotiation.cc:407] Connection header received
0928 06:36:05.515567 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:05.515593 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:05.515643 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:05.515713 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:05.516167 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.516720 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.517637 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.517853 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.517929 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:05.524676 (+ 6747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.3:43267: BlockingRecv error: recv got EOF from 127.5.22.3:43267 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 06:36:06 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:06 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:06.134053 14467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:06.128605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.1:58123)
0928 06:36:06.128893 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:06.128907 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:06.129005 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:06.129273 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:06.129275 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:06.129453 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:06.129589 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:06.129595 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.130398 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.130401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:06.130989 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:06.130999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.131484 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.131487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:06.131582 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:06.132100 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:06.132111 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:06.133934 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":74,"threads_started":1}
W20260928 06:36:06.134232 14468 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:06.128639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:58123 (local address 127.5.22.62:43529)
0928 06:36:06.128939 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:06.128943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:06.128957 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:36:06.129109 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:06.129111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:06.129156 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:06.129236 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:06.129690 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.130247 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.131126 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.131367 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.131475 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:06.134082 (+ 2607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.1:58123: BlockingRecv error: recv got EOF from 127.5.22.1:58123 (error 108)
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":123,"threads_started":1}
Sep 28 06:36:06 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:06 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:06.317685 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:06.312121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.2:36001)
0928 06:36:06.312412 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:06.312433 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:06.312532 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:06.312783 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:06.312785 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:06.312962 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:06.313060 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:06.313065 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.313783 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.313786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:06.314411 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:06.314420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.314873 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.314875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:06.314938 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:06.315677 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:06.315690 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:06.317558 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":104,"threads_started":1}
W20260928 06:36:06.317880 14468 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:06.312229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:36001 (local address 127.5.22.62:43529)
0928 06:36:06.312392 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:06.312396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:06.312505 (+ 109us) server_negotiation.cc:407] Connection header received
0928 06:36:06.312628 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:06.312631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:06.312674 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:06.312743 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:06.313162 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.313689 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.314521 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.314759 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.314853 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:06.317714 (+ 2861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.2:36001: BlockingRecv error: recv got EOF from 127.5.22.2:36001 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 06:36:06 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:06 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:06.531204 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:06.525417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.3:33203)
0928 06:36:06.525772 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:06.525789 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:06.525938 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:06.526216 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:06.526219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:06.526462 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:06.526564 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:06.526570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.527333 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.527340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:06.527904 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:06.527912 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.528384 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.528388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:06.528474 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:06.529017 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:06.529028 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:06.531089 (+ 2061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":137,"threads_started":1}
W20260928 06:36:06.531348 14468 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:06.525621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:33203 (local address 127.5.22.62:43529)
0928 06:36:06.525772 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:06.525779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:06.525877 (+ 98us) server_negotiation.cc:407] Connection header received
0928 06:36:06.526041 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:06.526044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:06.526104 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:06.526193 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:06.526688 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.527223 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.528025 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.528261 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.528372 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:06.531230 (+ 2858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.3:33203: BlockingRecv error: recv got EOF from 127.5.22.3:33203 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 06:36:07 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:07 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:07.140623 14471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:07.134916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.1:40721)
0928 06:36:07.135193 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:07.135211 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:07.135315 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:07.135654 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:07.135657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:07.135854 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:07.135952 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:07.135958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:07.136888 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:07.136891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:07.137465 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:07.137473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:07.137892 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:07.137895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:07.137954 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:07.138563 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:07.138577 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:07.140446 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":100,"threads_started":1}
W20260928 06:36:07.140697 14472 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:07.135024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:40721 (local address 127.5.22.62:43529)
0928 06:36:07.135347 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:07.135350 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:07.135383 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:36:07.135449 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:07.135452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:07.135505 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:07.135602 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:07.136112 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:07.136788 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:07.137575 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:07.137796 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:07.137872 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:07.140589 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.1:40721: BlockingRecv error: recv got EOF from 127.5.22.1:40721 (error 108)
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":105,"threads_started":1}
Sep 28 06:36:07 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:07 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:07.324676 14473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:07.318536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.2:60133)
0928 06:36:07.318852 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:07.318868 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:07.319016 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:07.319345 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:07.319348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:07.319558 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:07.319660 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:07.319666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:07.320440 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:07.320446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:07.321035 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:07.321043 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:07.321532 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:07.321536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:07.321634 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:07.322214 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:07.322245 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:07.324525 (+ 2280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":88,"threads_started":1}
W20260928 06:36:07.324803 14472 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:07.318645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:60133 (local address 127.5.22.62:43529)
0928 06:36:07.318818 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:07.318823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:07.318955 (+ 132us) server_negotiation.cc:407] Connection header received
0928 06:36:07.319145 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:07.319148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:07.319195 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:07.319322 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:07.319785 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:07.320326 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:07.321169 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:07.321403 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:07.321530 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:07.324684 (+ 3154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.2:60133: BlockingRecv error: recv got EOF from 127.5.22.2:60133 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 06:36:07 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:07 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.3@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:07.538010 14474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:07.532184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.3:41831)
0928 06:36:07.532558 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:07.532582 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:07.532717 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:07.533050 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:07.533053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:07.533333 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:07.533472 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:07.533479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:07.534492 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:07.534494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:07.535056 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:07.535064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:07.535486 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:07.535489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:07.535563 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:07.536060 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:07.536073 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:07.537876 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":92,"threads_started":1}
W20260928 06:36:07.538156 14472 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:07.532284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:41831 (local address 127.5.22.62:43529)
0928 06:36:07.532510 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:07.532516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:07.532664 (+ 148us) server_negotiation.cc:407] Connection header received
0928 06:36:07.532842 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:07.532847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:07.532902 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:07.533030 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:07.533596 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:07.534397 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:07.535158 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:07.535364 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:07.535439 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:07.538031 (+ 2592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.3:41831: BlockingRecv error: recv got EOF from 127.5.22.3:41831 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 28 06:36:08 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:08 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.1@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:08.147315 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:08.141609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.1:59693)
0928 06:36:08.141896 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:08.141911 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:08.142001 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:08.142352 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:08.142358 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:08.142611 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:08.142716 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:08.142722 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:08.143491 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:08.143494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:08.144145 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:08.144155 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:08.144601 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:08.144604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:08.144670 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:08.145298 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:08.145311 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:08.147197 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":90,"threads_started":1}
W20260928 06:36:08.147460 14476 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:08.141675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:59693 (local address 127.5.22.62:43529)
0928 06:36:08.141965 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:08.141970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:08.141984 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:36:08.142098 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:08.142100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:08.142160 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:08.142270 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:08.142833 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:08.143382 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:08.144261 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:08.144486 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:08.144566 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:08.147325 (+ 2759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.1:59693: BlockingRecv error: recv got EOF from 127.5.22.1:59693 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":79,"threads_started":1}
Sep 28 06:36:08 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:08 dist-test-slave-j0qf krb5kdc[13418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.22.2@KRBTEST.COM for oryx/127.5.22.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:08.331707 14477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:08.325858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43529 (local address 127.5.22.2:58995)
0928 06:36:08.326270 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:08.326283 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:08.326482 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:08.326824 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:08.326826 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:08.327040 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:08.327128 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:08.327134 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:08.327892 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:08.327896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:08.328464 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:08.328472 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:08.328940 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:08.328944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:08.329033 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:08.329589 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:08.329601 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:08.331511 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.22.62:43529: Server oryx/127.5.22.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":140,"threads_started":1}
W20260928 06:36:08.331769 14476 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:08.326050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:58995 (local address 127.5.22.62:43529)
0928 06:36:08.326242 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:08.326248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:08.326434 (+ 186us) server_negotiation.cc:407] Connection header received
0928 06:36:08.326662 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:08.326664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:08.326711 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:08.326800 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:08.327245 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:08.327783 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:08.328592 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:08.328825 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:08.328937 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:08.331655 (+ 2718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.22.2:58995: BlockingRecv error: recv got EOF from 127.5.22.2:58995 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 06:36:08.420007 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 14001
I20260928 06:36:08.425894 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 14153
I20260928 06:36:08.431053 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 14285
I20260928 06:36:08.436314 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 13931
2026-09-28T06:36:08Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18019 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T06:36:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:36:08Z Disabled control of system clock
I20260928 06:36:08.467365 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:35389
--webserver_interface=127.5.22.62
--webserver_port=0
--builtin_ntp_servers=127.5.22.20:43681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:35389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 06:36:08.581060 14481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:08.581326 14481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:08.581359 14481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:08.585343 14481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:36:08.585405 14481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:08.585420 14481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:08.585438 14481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:36:08.585453 14481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:36:08.590215 14481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:43681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:35389
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:35389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:36:08.591298 14481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:08.592720 14481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:08.601236 14486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:08.601426 14481 server_base.cc:1034] running on GCE node
W20260928 06:36:08.601233 14489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:08.601279 14487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:08.602057 14481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:08.603220 14481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:08.604380 14481 hybrid_clock.cc:648] HybridClock initialized: now 1790577368604325 us; error 66 us; skew 500 ppm
I20260928 06:36:08.606206 14481 webserver.cc:460] Webserver started at http://127.5.22.62:37617/ using document root <none> and password file <none>
I20260928 06:36:08.606825 14481 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:08.606889 14481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:08.607111 14481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:08.609133 14481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "7422a54a451042ab9215bd219bdd26ff"
format_stamp: "Formatted at 2026-09-28 06:36:08 on dist-test-slave-j0qf"
server_key: "33b01554bede20a5899e687a9f02ce2e"
server_key_iv: "f60b08c788e398d95eec7e91e158010f"
server_key_version: "encryptionkey@0"
I20260928 06:36:08.609710 14481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "7422a54a451042ab9215bd219bdd26ff"
format_stamp: "Formatted at 2026-09-28 06:36:08 on dist-test-slave-j0qf"
server_key: "33b01554bede20a5899e687a9f02ce2e"
server_key_iv: "f60b08c788e398d95eec7e91e158010f"
server_key_version: "encryptionkey@0"
I20260928 06:36:08.613391 14481 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 06:36:08.616261 14495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:08.617609 14481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:36:08.617745 14481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "7422a54a451042ab9215bd219bdd26ff"
format_stamp: "Formatted at 2026-09-28 06:36:08 on dist-test-slave-j0qf"
server_key: "33b01554bede20a5899e687a9f02ce2e"
server_key_iv: "f60b08c788e398d95eec7e91e158010f"
server_key_version: "encryptionkey@0"
I20260928 06:36:08.617869 14481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:08.643406 14481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:08.644156 14481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:08.644366 14481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:08.651774 14481 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:35389
I20260928 06:36:08.651800 14547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:35389 every 8 connection(s)
I20260928 06:36:08.652786 14481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:36:08.653431 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 14481
I20260928 06:36:08.653529 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:36:08.653831 5208 external_mini_cluster.cc:1428] Setting key 199a3f7e94f40a8fa3b44250b528e404
I20260928 06:36:08.656503 14548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:08.661971 14548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7422a54a451042ab9215bd219bdd26ff: Bootstrap starting.
I20260928 06:36:08.663450 14551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:08.655394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40004 (local address 127.5.22.62:35389)
0928 06:36:08.656061 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:08.656078 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:08.656117 (+ 39us) server_negotiation.cc:407] Connection header received
0928 06:36:08.656833 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:08.656853 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:08.657243 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:08.657589 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:08.658028 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:08.659078 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:08.659953 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:08.660423 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:08.660614 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:08.661606 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:08.661611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:08.661622 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:36:08.662012 (+ 390us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:08.662069 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:36:08.662115 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:08.662218 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:08.662431 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 06:36:08.662748 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":153,"threads_started":1}
I20260928 06:36:08.664358 14548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7422a54a451042ab9215bd219bdd26ff: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:08.665233 14548 log.cc:824] T 00000000000000000000000000000000 P 7422a54a451042ab9215bd219bdd26ff: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:08.667107 14548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7422a54a451042ab9215bd219bdd26ff: No bootstrap required, opened a new log
I20260928 06:36:08.670495 14548 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7422a54a451042ab9215bd219bdd26ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7422a54a451042ab9215bd219bdd26ff" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35389 } }
I20260928 06:36:08.670701 14548 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7422a54a451042ab9215bd219bdd26ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:08.670734 14548 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7422a54a451042ab9215bd219bdd26ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7422a54a451042ab9215bd219bdd26ff, State: Initialized, Role: FOLLOWER
I20260928 06:36:08.671075 14548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7422a54a451042ab9215bd219bdd26ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7422a54a451042ab9215bd219bdd26ff" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35389 } }
I20260928 06:36:08.671216 14548 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7422a54a451042ab9215bd219bdd26ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:36:08.671301 14548 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7422a54a451042ab9215bd219bdd26ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:36:08.671391 14548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7422a54a451042ab9215bd219bdd26ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:08.672271 14548 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7422a54a451042ab9215bd219bdd26ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7422a54a451042ab9215bd219bdd26ff" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35389 } }
I20260928 06:36:08.672616 14548 leader_election.cc:302] T 00000000000000000000000000000000 P 7422a54a451042ab9215bd219bdd26ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7422a54a451042ab9215bd219bdd26ff; no voters:
I20260928 06:36:08.672904 14548 leader_election.cc:288] T 00000000000000000000000000000000 P 7422a54a451042ab9215bd219bdd26ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:36:08.673017 14553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7422a54a451042ab9215bd219bdd26ff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:36:08.673276 14553 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7422a54a451042ab9215bd219bdd26ff [term 1 LEADER]: Becoming Leader. State: Replica: 7422a54a451042ab9215bd219bdd26ff, State: Running, Role: LEADER
I20260928 06:36:08.673631 14553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7422a54a451042ab9215bd219bdd26ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7422a54a451042ab9215bd219bdd26ff" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35389 } }
I20260928 06:36:08.674116 14548 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7422a54a451042ab9215bd219bdd26ff [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:36:08.675302 14554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7422a54a451042ab9215bd219bdd26ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7422a54a451042ab9215bd219bdd26ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7422a54a451042ab9215bd219bdd26ff" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35389 } } }
I20260928 06:36:08.675441 14554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7422a54a451042ab9215bd219bdd26ff [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:08.675640 14555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7422a54a451042ab9215bd219bdd26ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7422a54a451042ab9215bd219bdd26ff. Latest consensus state: current_term: 1 leader_uuid: "7422a54a451042ab9215bd219bdd26ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7422a54a451042ab9215bd219bdd26ff" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35389 } } }
I20260928 06:36:08.676148 14555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7422a54a451042ab9215bd219bdd26ff [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:08.676560 14563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:36:08.679246 14563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:36:08.684458 14563 catalog_manager.cc:1347] Generated new cluster ID: 293065045d3545fc99e99a86589948c7
I20260928 06:36:08.684530 14563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:36:08.699205 14563 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:36:08.700147 14563 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:36:08.708200 14563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7422a54a451042ab9215bd219bdd26ff: Generated new TSK 0
I20260928 06:36:08.708918 14563 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:36:08.717085 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:35389
--builtin_ntp_servers=127.5.22.20:43681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 06:36:08.831387 14572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:08.831658 14572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:08.831750 14572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:08.836098 14572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:08.836158 14572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:08.836256 14572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:36:08.841084 14572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:43681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:35389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:36:08.842254 14572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:08.843776 14572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:08.853526 14577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:08.853549 14578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:08.853526 14580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:08.853896 14572 server_base.cc:1034] running on GCE node
I20260928 06:36:08.854362 14572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:08.855007 14572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:08.856199 14572 hybrid_clock.cc:648] HybridClock initialized: now 1790577368856145 us; error 65 us; skew 500 ppm
I20260928 06:36:08.858561 14572 webserver.cc:460] Webserver started at http://127.5.22.1:38299/ using document root <none> and password file <none>
I20260928 06:36:08.859207 14572 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:08.859301 14572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:08.859548 14572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:08.861620 14572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "45876147fae44d96bb305d55106afcc0"
format_stamp: "Formatted at 2026-09-28 06:36:08 on dist-test-slave-j0qf"
server_key: "f628f356f386898779c68baa38ce8e23"
server_key_iv: "f1edfbae91beec584266624566e611ca"
server_key_version: "encryptionkey@0"
I20260928 06:36:08.862191 14572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "45876147fae44d96bb305d55106afcc0"
format_stamp: "Formatted at 2026-09-28 06:36:08 on dist-test-slave-j0qf"
server_key: "f628f356f386898779c68baa38ce8e23"
server_key_iv: "f1edfbae91beec584266624566e611ca"
server_key_version: "encryptionkey@0"
I20260928 06:36:08.865983 14572 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 06:36:08.868611 14586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:08.870082 14572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:36:08.870237 14572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "45876147fae44d96bb305d55106afcc0"
format_stamp: "Formatted at 2026-09-28 06:36:08 on dist-test-slave-j0qf"
server_key: "f628f356f386898779c68baa38ce8e23"
server_key_iv: "f1edfbae91beec584266624566e611ca"
server_key_version: "encryptionkey@0"
I20260928 06:36:08.870445 14572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:08.894479 14572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:08.895258 14572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:08.895483 14572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:08.896281 14572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:08.897358 14572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:08.897409 14572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:08.897468 14572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:08.897490 14572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:08.907265 14572 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:37701
I20260928 06:36:08.907298 14699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:37701 every 8 connection(s)
I20260928 06:36:08.908313 14572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:36:08.914211 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 14572
I20260928 06:36:08.914453 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:36:08.914785 5208 external_mini_cluster.cc:1428] Setting key dc02d97cd9aca3ad53eca18012e4a409
I20260928 06:36:08.917146 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:35389
--builtin_ntp_servers=127.5.22.20:43681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 06:36:08.919202 14551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:08.910533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:58395 (local address 127.5.22.62:35389)
0928 06:36:08.910730 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:08.910735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:08.911925 (+ 1190us) server_negotiation.cc:407] Connection header received
0928 06:36:08.913373 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:08.913376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:08.913442 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:08.913532 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:08.914639 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:08.915370 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:08.916904 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:08.917150 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:08.917268 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:08.918574 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:08.918580 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:08.918584 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:36:08.918606 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:08.918637 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:36:08.918651 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:08.918734 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:08.918928 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 06:36:08.919007 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 06:36:08.920080 14702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:08.911072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:35389 (local address 127.5.22.1:58395)
0928 06:36:08.911756 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:08.911800 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:08.913143 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:08.913735 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:08.913743 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:08.913833 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:08.914428 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:08.914454 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:08.915488 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:08.915492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:08.916776 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:08.916785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:08.917310 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:08.917315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:08.917500 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:08.918176 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:36:08.918177 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:08.918219 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:36:08.918291 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:08.918758 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:08.918766 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:08.918772 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:36:08.918882 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 06:36:08.919139 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":121,"threads_started":1}
I20260928 06:36:08.921651 14700 heartbeater.cc:347] Connected to a master server at 127.5.22.62:35389
I20260928 06:36:08.921896 14700 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:08.922566 14700 heartbeater.cc:511] Master 127.5.22.62:35389 requested a full tablet report, sending...
I20260928 06:36:08.924162 14512 ts_manager.cc:194] Registered new tserver with Master: 45876147fae44d96bb305d55106afcc0 (127.5.22.1:37701)
I20260928 06:36:08.925328 14512 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.22.1:58395
W20260928 06:36:09.037204 14703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:09.037467 14703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:09.037504 14703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:09.041641 14703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:09.041705 14703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:09.041795 14703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:36:09.047358 14703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:43681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:35389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:36:09.048456 14703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:09.049898 14703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:09.059906 14708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:09.059912 14709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:09.059950 14703 server_base.cc:1034] running on GCE node
W20260928 06:36:09.059912 14711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:09.060606 14703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:09.061187 14703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:09.062387 14703 hybrid_clock.cc:648] HybridClock initialized: now 1790577369062369 us; error 30 us; skew 500 ppm
I20260928 06:36:09.064316 14703 webserver.cc:460] Webserver started at http://127.5.22.2:35027/ using document root <none> and password file <none>
I20260928 06:36:09.064895 14703 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:09.064960 14703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:09.065232 14703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:09.067375 14703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "923446fee65b471683f21cc1b41d03a7"
format_stamp: "Formatted at 2026-09-28 06:36:09 on dist-test-slave-j0qf"
server_key: "0161260ddd16ccd63f6fae3fe0d9289c"
server_key_iv: "83ed132ee8df575a973046dfa85c3d56"
server_key_version: "encryptionkey@0"
I20260928 06:36:09.067961 14703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "923446fee65b471683f21cc1b41d03a7"
format_stamp: "Formatted at 2026-09-28 06:36:09 on dist-test-slave-j0qf"
server_key: "0161260ddd16ccd63f6fae3fe0d9289c"
server_key_iv: "83ed132ee8df575a973046dfa85c3d56"
server_key_version: "encryptionkey@0"
I20260928 06:36:09.071846 14703 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 06:36:09.074545 14717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.075925 14703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:36:09.076066 14703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "923446fee65b471683f21cc1b41d03a7"
format_stamp: "Formatted at 2026-09-28 06:36:09 on dist-test-slave-j0qf"
server_key: "0161260ddd16ccd63f6fae3fe0d9289c"
server_key_iv: "83ed132ee8df575a973046dfa85c3d56"
server_key_version: "encryptionkey@0"
I20260928 06:36:09.076172 14703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:09.130887 14703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:09.131651 14703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:09.131852 14703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:09.132473 14703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:09.133417 14703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:09.133463 14703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.133503 14703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:09.133520 14703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.142472 14703 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:41285
I20260928 06:36:09.142546 14830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:41285 every 8 connection(s)
I20260928 06:36:09.143426 14703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:36:09.144995 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 14703
I20260928 06:36:09.145138 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:36:09.145488 5208 external_mini_cluster.cc:1428] Setting key 2b4b0c27f73ce6fc15458415caf302b6
I20260928 06:36:09.147418 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:35389
--builtin_ntp_servers=127.5.22.20:43681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 06:36:09.153616 14551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:09.145287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:60849 (local address 127.5.22.62:35389)
0928 06:36:09.145455 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:09.145459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:09.146616 (+ 1157us) server_negotiation.cc:407] Connection header received
0928 06:36:09.148057 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:09.148060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:09.148111 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:09.148186 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:09.149158 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.149683 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.151003 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.151443 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.151604 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:09.152925 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:09.152931 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:09.152936 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:36:09.152954 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:09.152982 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:36:09.152997 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:09.153094 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:09.153273 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 06:36:09.153356 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 06:36:09.154453 14833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:09.145726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:35389 (local address 127.5.22.2:60849)
0928 06:36:09.146452 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:09.146505 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:09.147837 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:09.148348 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:09.148356 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:09.148437 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:09.148943 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:09.148958 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.149850 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.149853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:09.150844 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:09.150854 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.151765 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.151768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:09.151929 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:09.152651 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:36:09.152653 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:09.152693 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:36:09.152760 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:09.153118 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:09.153126 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:09.153131 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:36:09.153241 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 06:36:09.153461 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":132,"threads_started":1}
I20260928 06:36:09.155781 14831 heartbeater.cc:347] Connected to a master server at 127.5.22.62:35389
I20260928 06:36:09.156032 14831 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:09.156558 14831 heartbeater.cc:511] Master 127.5.22.62:35389 requested a full tablet report, sending...
I20260928 06:36:09.157627 14512 ts_manager.cc:194] Registered new tserver with Master: 923446fee65b471683f21cc1b41d03a7 (127.5.22.2:41285)
I20260928 06:36:09.158507 14512 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.22.2:60849
W20260928 06:36:09.258443 14834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:09.258700 14834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:09.258738 14834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:09.262921 14834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:09.262982 14834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:09.263038 14834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:36:09.267846 14834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:43681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:35389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:36:09.268895 14834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:09.270206 14834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:36:09.280112 14834 server_base.cc:1034] running on GCE node
W20260928 06:36:09.279991 14840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:09.280026 14839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:09.279991 14842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:09.280859 14834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:09.281577 14834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:09.282757 14834 hybrid_clock.cc:648] HybridClock initialized: now 1790577369282743 us; error 25 us; skew 500 ppm
I20260928 06:36:09.284577 14834 webserver.cc:460] Webserver started at http://127.5.22.3:45139/ using document root <none> and password file <none>
I20260928 06:36:09.285157 14834 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:09.285250 14834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:09.285481 14834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:09.287588 14834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "fa007d5e84194f90905ba968064ace1a"
format_stamp: "Formatted at 2026-09-28 06:36:09 on dist-test-slave-j0qf"
server_key: "452f3dd91d3593f11cb3166bf70b16a0"
server_key_iv: "b2101ceb924698dfa76ed5eb7ad42b48"
server_key_version: "encryptionkey@0"
I20260928 06:36:09.288144 14834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "fa007d5e84194f90905ba968064ace1a"
format_stamp: "Formatted at 2026-09-28 06:36:09 on dist-test-slave-j0qf"
server_key: "452f3dd91d3593f11cb3166bf70b16a0"
server_key_iv: "b2101ceb924698dfa76ed5eb7ad42b48"
server_key_version: "encryptionkey@0"
I20260928 06:36:09.291826 14834 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:36:09.294539 14848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.295761 14834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:36:09.295893 14834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "fa007d5e84194f90905ba968064ace1a"
format_stamp: "Formatted at 2026-09-28 06:36:09 on dist-test-slave-j0qf"
server_key: "452f3dd91d3593f11cb3166bf70b16a0"
server_key_iv: "b2101ceb924698dfa76ed5eb7ad42b48"
server_key_version: "encryptionkey@0"
I20260928 06:36:09.296006 14834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:09.316684 14834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:09.317390 14834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:09.317610 14834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:09.318205 14834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:09.319171 14834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:09.319242 14834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.319314 14834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:09.319362 14834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.328516 14834 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:33285
I20260928 06:36:09.328540 14961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:33285 every 8 connection(s)
I20260928 06:36:09.329429 14834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:36:09.333963 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 14834
I20260928 06:36:09.334103 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
I20260928 06:36:09.334527 5208 external_mini_cluster.cc:1428] Setting key 6f0517f3371fb9db36993c41dd213c8a
I20260928 06:36:09.339304 14551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:09.331395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:39249 (local address 127.5.22.62:35389)
0928 06:36:09.331539 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:09.331543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:09.332653 (+ 1110us) server_negotiation.cc:407] Connection header received
0928 06:36:09.334087 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:09.334090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:09.334152 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:09.334267 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:09.335447 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.336193 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.337476 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.337726 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.337808 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:09.338882 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:09.338885 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:09.338887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:36:09.338899 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:09.338918 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:36:09.338926 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:09.338983 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:09.339145 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 06:36:09.339195 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:36:09.340166 14964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:09.331772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:35389 (local address 127.5.22.3:39249)
0928 06:36:09.332485 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:09.332527 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:09.333858 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:09.334490 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:09.334498 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:09.334585 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:09.335275 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:09.335295 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.336329 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.336333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:09.337365 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:09.337375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.337862 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.337865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:09.338001 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:09.338645 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:36:09.338646 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:09.338684 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:36:09.338745 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:09.338998 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:09.339003 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:09.339007 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:36:09.339134 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 06:36:09.339373 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":122,"threads_started":1}
I20260928 06:36:09.341231 14962 heartbeater.cc:347] Connected to a master server at 127.5.22.62:35389
I20260928 06:36:09.341465 14962 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:09.342005 14962 heartbeater.cc:511] Master 127.5.22.62:35389 requested a full tablet report, sending...
I20260928 06:36:09.343039 14512 ts_manager.cc:194] Registered new tserver with Master: fa007d5e84194f90905ba968064ace1a (127.5.22.3:33285)
I20260928 06:36:09.343580 14512 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.22.3:39249
I20260928 06:36:09.348431 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 06:36:09.351207 14551 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:09.350355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40016 (local address 127.5.22.62:35389)
0928 06:36:09.350469 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:09.350473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:09.350659 (+ 186us) server_negotiation.cc:407] Connection header received
0928 06:36:09.350756 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:09.350758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:09.350798 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:09.350857 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:09.351063 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40016: BlockingRecv error: recv got EOF from 127.0.0.1:40016 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:36:09.351493 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 14572
I20260928 06:36:09.357189 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 14703
I20260928 06:36:09.362571 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 14834
I20260928 06:36:09.367842 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 14481
2026-09-28T06:36:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (930 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T06:36:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:36:09Z Disabled control of system clock
I20260928 06:36:09.397578 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:37805
--webserver_interface=127.5.22.62
--webserver_port=0
--builtin_ntp_servers=127.5.22.20:39683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:37805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 06:36:09.509285 14973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:09.509665 14973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:09.509744 14973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:09.513747 14973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:36:09.513840 14973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:09.513871 14973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:09.513911 14973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:36:09.513943 14973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:36:09.518711 14973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:39683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:37805
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:37805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:36:09.519845 14973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:09.521260 14973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:09.529839 14981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:09.530022 14973 server_base.cc:1034] running on GCE node
W20260928 06:36:09.529843 14978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:09.529843 14979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:09.530802 14973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:09.531996 14973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:09.533190 14973 hybrid_clock.cc:648] HybridClock initialized: now 1790577369533174 us; error 27 us; skew 500 ppm
I20260928 06:36:09.535176 14973 webserver.cc:460] Webserver started at http://127.5.22.62:38311/ using document root <none> and password file <none>
I20260928 06:36:09.535758 14973 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:09.535852 14973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:09.536085 14973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:09.538128 14973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "a5db700d1aee4c37940ea3fa3388ba15"
format_stamp: "Formatted at 2026-09-28 06:36:09 on dist-test-slave-j0qf"
server_key: "e7b4bd4e9f7bcf54e7d1f0606cf5a99e"
server_key_iv: "d61a712a0ce59b3aa11859f4821ba415"
server_key_version: "encryptionkey@0"
I20260928 06:36:09.538757 14973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "a5db700d1aee4c37940ea3fa3388ba15"
format_stamp: "Formatted at 2026-09-28 06:36:09 on dist-test-slave-j0qf"
server_key: "e7b4bd4e9f7bcf54e7d1f0606cf5a99e"
server_key_iv: "d61a712a0ce59b3aa11859f4821ba415"
server_key_version: "encryptionkey@0"
I20260928 06:36:09.542546 14973 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 06:36:09.545161 14987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.546615 14973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:36:09.546763 14973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "a5db700d1aee4c37940ea3fa3388ba15"
format_stamp: "Formatted at 2026-09-28 06:36:09 on dist-test-slave-j0qf"
server_key: "e7b4bd4e9f7bcf54e7d1f0606cf5a99e"
server_key_iv: "d61a712a0ce59b3aa11859f4821ba415"
server_key_version: "encryptionkey@0"
I20260928 06:36:09.546888 14973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:09.551831 14973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:09.552493 14973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:09.552696 14973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:09.559995 14973 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:37805
I20260928 06:36:09.559998 15039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:37805 every 8 connection(s)
I20260928 06:36:09.561038 14973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:36:09.563257 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 14973
I20260928 06:36:09.563380 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:36:09.563692 5208 external_mini_cluster.cc:1428] Setting key cd9e9764b551e57ecdfbda4a46df83b4
I20260928 06:36:09.564839 15040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:09.570192 15043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:09.565266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43118 (local address 127.5.22.62:37805)
0928 06:36:09.565967 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:09.565986 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:09.566030 (+ 44us) server_negotiation.cc:407] Connection header received
0928 06:36:09.566934 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:09.566959 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:09.567403 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:09.567784 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:09.568675 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:09.568681 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:09.568695 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:36:09.568708 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:09.568778 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:36:09.568801 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:09.568945 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:09.569052 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 06:36:09.569292 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":138,"threads_started":1}
I20260928 06:36:09.571965 15040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5db700d1aee4c37940ea3fa3388ba15: Bootstrap starting.
I20260928 06:36:09.574205 15040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5db700d1aee4c37940ea3fa3388ba15: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:09.574949 15040 log.cc:824] T 00000000000000000000000000000000 P a5db700d1aee4c37940ea3fa3388ba15: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:09.576750 15040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5db700d1aee4c37940ea3fa3388ba15: No bootstrap required, opened a new log
I20260928 06:36:09.580229 15040 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5db700d1aee4c37940ea3fa3388ba15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5db700d1aee4c37940ea3fa3388ba15" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 37805 } }
I20260928 06:36:09.580441 15040 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5db700d1aee4c37940ea3fa3388ba15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:09.580475 15040 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5db700d1aee4c37940ea3fa3388ba15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5db700d1aee4c37940ea3fa3388ba15, State: Initialized, Role: FOLLOWER
I20260928 06:36:09.580878 15040 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5db700d1aee4c37940ea3fa3388ba15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5db700d1aee4c37940ea3fa3388ba15" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 37805 } }
I20260928 06:36:09.580996 15040 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5db700d1aee4c37940ea3fa3388ba15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:36:09.581036 15040 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5db700d1aee4c37940ea3fa3388ba15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:36:09.581108 15040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5db700d1aee4c37940ea3fa3388ba15 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:09.581890 15040 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5db700d1aee4c37940ea3fa3388ba15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5db700d1aee4c37940ea3fa3388ba15" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 37805 } }
I20260928 06:36:09.582273 15040 leader_election.cc:302] T 00000000000000000000000000000000 P a5db700d1aee4c37940ea3fa3388ba15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5db700d1aee4c37940ea3fa3388ba15; no voters:
I20260928 06:36:09.582662 15040 leader_election.cc:288] T 00000000000000000000000000000000 P a5db700d1aee4c37940ea3fa3388ba15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:36:09.582753 15045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5db700d1aee4c37940ea3fa3388ba15 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:36:09.583031 15045 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5db700d1aee4c37940ea3fa3388ba15 [term 1 LEADER]: Becoming Leader. State: Replica: a5db700d1aee4c37940ea3fa3388ba15, State: Running, Role: LEADER
I20260928 06:36:09.583372 15045 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5db700d1aee4c37940ea3fa3388ba15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5db700d1aee4c37940ea3fa3388ba15" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 37805 } }
I20260928 06:36:09.583895 15040 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5db700d1aee4c37940ea3fa3388ba15 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:36:09.584962 15046 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5db700d1aee4c37940ea3fa3388ba15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5db700d1aee4c37940ea3fa3388ba15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5db700d1aee4c37940ea3fa3388ba15" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 37805 } } }
I20260928 06:36:09.585086 15046 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5db700d1aee4c37940ea3fa3388ba15 [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:09.585417 15047 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5db700d1aee4c37940ea3fa3388ba15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5db700d1aee4c37940ea3fa3388ba15. Latest consensus state: current_term: 1 leader_uuid: "a5db700d1aee4c37940ea3fa3388ba15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5db700d1aee4c37940ea3fa3388ba15" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 37805 } } }
I20260928 06:36:09.585508 15054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:36:09.585531 15047 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5db700d1aee4c37940ea3fa3388ba15 [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:09.588692 15054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:36:09.594081 15054 catalog_manager.cc:1347] Generated new cluster ID: 70db470bdf7047868ca8356c707b52e5
I20260928 06:36:09.594192 15054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:36:09.613052 15054 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:36:09.613436 15054 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:36:09.620754 15054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5db700d1aee4c37940ea3fa3388ba15: Generated new TSK 0
I20260928 06:36:09.621522 15054 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:36:09.634258 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:37805
--builtin_ntp_servers=127.5.22.20:39683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 06:36:09.745631 15064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:09.745923 15064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:09.745965 15064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:09.750152 15064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:09.750213 15064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:09.750317 15064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:36:09.755440 15064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:39683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:37805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:36:09.756579 15064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:09.758018 15064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:09.767913 15070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:09.767913 15072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:09.767932 15069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:09.768376 15064 server_base.cc:1034] running on GCE node
I20260928 06:36:09.768730 15064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:09.769387 15064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:09.770542 15064 hybrid_clock.cc:648] HybridClock initialized: now 1790577369770531 us; error 27 us; skew 500 ppm
I20260928 06:36:09.772596 15064 webserver.cc:460] Webserver started at http://127.5.22.1:34809/ using document root <none> and password file <none>
I20260928 06:36:09.773129 15064 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:09.773186 15064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:09.773342 15064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:09.775393 15064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "82fe80a2c9344c76b3f70022a8c15cac"
format_stamp: "Formatted at 2026-09-28 06:36:09 on dist-test-slave-j0qf"
server_key: "da0e65af33e8c9db2af6805d2bd8e62f"
server_key_iv: "f9485ec631ef03b6a82cc1fc4f798a2a"
server_key_version: "encryptionkey@0"
I20260928 06:36:09.775915 15064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "82fe80a2c9344c76b3f70022a8c15cac"
format_stamp: "Formatted at 2026-09-28 06:36:09 on dist-test-slave-j0qf"
server_key: "da0e65af33e8c9db2af6805d2bd8e62f"
server_key_iv: "f9485ec631ef03b6a82cc1fc4f798a2a"
server_key_version: "encryptionkey@0"
I20260928 06:36:09.779495 15064 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:36:09.781893 15078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.783215 15064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:36:09.783309 15064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "82fe80a2c9344c76b3f70022a8c15cac"
format_stamp: "Formatted at 2026-09-28 06:36:09 on dist-test-slave-j0qf"
server_key: "da0e65af33e8c9db2af6805d2bd8e62f"
server_key_iv: "f9485ec631ef03b6a82cc1fc4f798a2a"
server_key_version: "encryptionkey@0"
I20260928 06:36:09.783401 15064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:09.788120 15064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:09.788699 15064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:09.788877 15064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:09.789384 15064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:09.790218 15064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:09.790263 15064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.790299 15064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:09.790375 15064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.799245 15064 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:37907
I20260928 06:36:09.799260 15191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:37907 every 8 connection(s)
I20260928 06:36:09.800143 15064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:36:09.800819 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 15064
I20260928 06:36:09.800974 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:36:09.801258 5208 external_mini_cluster.cc:1428] Setting key f0244f8519c2e3f100dcaa7701f2cc05
I20260928 06:36:09.803630 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:37805
--builtin_ntp_servers=127.5.22.20:39683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 06:36:09.806865 15043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:09.802036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:42913 (local address 127.5.22.62:37805)
0928 06:36:09.802182 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:09.802186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:09.803509 (+ 1323us) server_negotiation.cc:407] Connection header received
0928 06:36:09.805109 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:09.805113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:09.805154 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:09.805233 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:09.806384 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:09.806388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:09.806391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:36:09.806403 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:09.806426 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:36:09.806437 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:09.806538 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:09.806686 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 06:36:09.806731 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 06:36:09.807715 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:09.802544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:37805 (local address 127.5.22.1:42913)
0928 06:36:09.803337 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:09.803392 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:09.804903 (+ 1511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:09.805399 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:09.805406 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:09.805494 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:09.806100 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:36:09.806102 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:09.806143 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:36:09.806210 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:09.806532 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:09.806541 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:09.806549 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 06:36:09.806733 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 06:36:09.806980 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":129,"threads_started":1}
I20260928 06:36:09.809031 15192 heartbeater.cc:347] Connected to a master server at 127.5.22.62:37805
I20260928 06:36:09.809296 15192 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:09.809827 15192 heartbeater.cc:511] Master 127.5.22.62:37805 requested a full tablet report, sending...
I20260928 06:36:09.811702 15004 ts_manager.cc:194] Registered new tserver with Master: 82fe80a2c9344c76b3f70022a8c15cac (127.5.22.1:37907)
W20260928 06:36:09.919445 15195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:09.919715 15195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:09.919756 15195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:09.923758 15195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:09.923811 15195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:09.923868 15195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:36:09.928632 15195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:39683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:37805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:36:09.929844 15195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:09.931675 15195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:09.941742 15203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:09.941890 15195 server_base.cc:1034] running on GCE node
W20260928 06:36:09.941736 15200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:09.941738 15201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:09.942464 15195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:09.943138 15195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:09.944350 15195 hybrid_clock.cc:648] HybridClock initialized: now 1790577369944342 us; error 37 us; skew 500 ppm
I20260928 06:36:09.946427 15195 webserver.cc:460] Webserver started at http://127.5.22.2:35911/ using document root <none> and password file <none>
I20260928 06:36:09.947036 15195 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:09.947134 15195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:09.947350 15195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:09.949409 15195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "45aa5182dc5c43b287ee047332f9cef8"
format_stamp: "Formatted at 2026-09-28 06:36:09 on dist-test-slave-j0qf"
server_key: "27c3dad7637ed0518e5eaf3237607e6f"
server_key_iv: "14402627b44b0ce90c7c6eab8501a462"
server_key_version: "encryptionkey@0"
I20260928 06:36:09.949981 15195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "45aa5182dc5c43b287ee047332f9cef8"
format_stamp: "Formatted at 2026-09-28 06:36:09 on dist-test-slave-j0qf"
server_key: "27c3dad7637ed0518e5eaf3237607e6f"
server_key_iv: "14402627b44b0ce90c7c6eab8501a462"
server_key_version: "encryptionkey@0"
I20260928 06:36:09.953909 15195 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260928 06:36:09.956450 15209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.957710 15195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:36:09.957841 15195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "45aa5182dc5c43b287ee047332f9cef8"
format_stamp: "Formatted at 2026-09-28 06:36:09 on dist-test-slave-j0qf"
server_key: "27c3dad7637ed0518e5eaf3237607e6f"
server_key_iv: "14402627b44b0ce90c7c6eab8501a462"
server_key_version: "encryptionkey@0"
I20260928 06:36:09.957952 15195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:09.962491 15195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:09.963080 15195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:09.963261 15195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:09.963789 15195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:09.964655 15195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:09.964725 15195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.964788 15195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:09.964840 15195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.973568 15195 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:45189
I20260928 06:36:09.973596 15322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:45189 every 8 connection(s)
I20260928 06:36:09.974483 15195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:36:09.980334 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 15195
I20260928 06:36:09.980479 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:36:09.980688 15043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:09.976289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:53841 (local address 127.5.22.62:37805)
0928 06:36:09.976434 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:09.976438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:09.977420 (+ 982us) server_negotiation.cc:407] Connection header received
0928 06:36:09.978812 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:09.978816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:09.978865 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:09.978973 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:09.979994 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:09.979998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:09.980001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:36:09.980013 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:09.980059 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:36:09.980078 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:09.980188 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:09.980372 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 06:36:09.980422 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:36:09.980979 5208 external_mini_cluster.cc:1428] Setting key 0de9f0fd4954fa7ba47485181d4a5445
I20260928 06:36:09.981735 15325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:09.976639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:37805 (local address 127.5.22.2:53841)
0928 06:36:09.977273 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:09.977312 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:09.978602 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:09.979136 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:09.979142 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:09.979227 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:09.979744 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:36:09.979745 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:09.979783 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:36:09.979848 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:09.980204 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:09.980211 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:09.980216 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:36:09.980604 (+ 388us) client_negotiation.cc:240] Negotiation successful
0928 06:36:09.980849 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":117,"threads_started":1}
I20260928 06:36:09.982903 15323 heartbeater.cc:347] Connected to a master server at 127.5.22.62:37805
I20260928 06:36:09.983142 15323 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:09.983269 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:37805
--builtin_ntp_servers=127.5.22.20:39683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 06:36:09.983593 15323 heartbeater.cc:511] Master 127.5.22.62:37805 requested a full tablet report, sending...
I20260928 06:36:09.984594 15004 ts_manager.cc:194] Registered new tserver with Master: 45aa5182dc5c43b287ee047332f9cef8 (127.5.22.2:45189)
W20260928 06:36:10.096837 15326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:10.097115 15326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:10.097155 15326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:10.101192 15326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:10.101253 15326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:10.101308 15326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:36:10.105984 15326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:39683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:37805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:36:10.107085 15326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:10.108444 15326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:10.118253 15334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:10.118453 15326 server_base.cc:1034] running on GCE node
W20260928 06:36:10.118228 15332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:10.118268 15331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:10.118996 15326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:10.119594 15326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:10.120828 15326 hybrid_clock.cc:648] HybridClock initialized: now 1790577370120779 us; error 31 us; skew 500 ppm
I20260928 06:36:10.122936 15326 webserver.cc:460] Webserver started at http://127.5.22.3:37683/ using document root <none> and password file <none>
I20260928 06:36:10.123482 15326 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:10.123543 15326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:10.123708 15326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:10.125705 15326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "68570cca03e44e52ace34ec6c23b6bea"
format_stamp: "Formatted at 2026-09-28 06:36:10 on dist-test-slave-j0qf"
server_key: "f5b48c3bf60aed45ea1d4fb3618591a7"
server_key_iv: "43e409234b2e7a98f8fd004ffc7e147b"
server_key_version: "encryptionkey@0"
I20260928 06:36:10.126209 15326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "68570cca03e44e52ace34ec6c23b6bea"
format_stamp: "Formatted at 2026-09-28 06:36:10 on dist-test-slave-j0qf"
server_key: "f5b48c3bf60aed45ea1d4fb3618591a7"
server_key_iv: "43e409234b2e7a98f8fd004ffc7e147b"
server_key_version: "encryptionkey@0"
I20260928 06:36:10.129834 15326 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 06:36:10.132515 15340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:10.133764 15326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 06:36:10.133885 15326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "68570cca03e44e52ace34ec6c23b6bea"
format_stamp: "Formatted at 2026-09-28 06:36:10 on dist-test-slave-j0qf"
server_key: "f5b48c3bf60aed45ea1d4fb3618591a7"
server_key_iv: "43e409234b2e7a98f8fd004ffc7e147b"
server_key_version: "encryptionkey@0"
I20260928 06:36:10.133993 15326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:10.138563 15326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:10.139163 15326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:10.139338 15326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:10.139889 15326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:10.140771 15326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:10.140817 15326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:10.140877 15326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:10.140909 15326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:10.149420 15326 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:41269
I20260928 06:36:10.149441 15453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:41269 every 8 connection(s)
I20260928 06:36:10.150364 15326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:36:10.156616 15043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:10.152312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:39313 (local address 127.5.22.62:37805)
0928 06:36:10.152536 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:10.152541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:10.153661 (+ 1120us) server_negotiation.cc:407] Connection header received
0928 06:36:10.155011 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:10.155014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:10.155052 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:10.155128 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:10.156181 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:10.156184 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:10.156186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:36:10.156197 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:10.156217 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:36:10.156225 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:10.156308 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:10.156464 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 06:36:10.156506 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 06:36:10.157397 15456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:10.152818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:37805 (local address 127.5.22.3:39313)
0928 06:36:10.153486 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:10.153527 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:10.154812 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:10.155289 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:10.155296 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:10.155380 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:10.155916 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:36:10.155918 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:10.155957 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:36:10.156022 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:10.156306 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:10.156312 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:10.156316 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:36:10.156441 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 06:36:10.156637 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":118,"threads_started":1}
I20260928 06:36:10.158561 15454 heartbeater.cc:347] Connected to a master server at 127.5.22.62:37805
I20260928 06:36:10.158794 15454 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:10.159186 15454 heartbeater.cc:511] Master 127.5.22.62:37805 requested a full tablet report, sending...
I20260928 06:36:10.159816 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 15326
I20260928 06:36:10.159946 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
I20260928 06:36:10.160236 5208 external_mini_cluster.cc:1428] Setting key df9ea611dc20c76fc03765994bafbb8d
I20260928 06:36:10.160704 15004 ts_manager.cc:194] Registered new tserver with Master: 68570cca03e44e52ace34ec6c23b6bea (127.5.22.3:41269)
I20260928 06:36:10.162379 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 06:36:10.165190 15043 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:10.164219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43124 (local address 127.5.22.62:37805)
0928 06:36:10.164311 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:10.164314 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:10.164558 (+ 244us) server_negotiation.cc:407] Connection header received
0928 06:36:10.164709 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:10.164711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:10.164744 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:10.164833 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:10.165072 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43124: BlockingRecv error: recv got EOF from 127.0.0.1:43124 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:36:10.165323 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 15064
I20260928 06:36:10.171240 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 15195
I20260928 06:36:10.176990 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 15326
I20260928 06:36:10.182366 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 14973
2026-09-28T06:36:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (813 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:36:10 dist-test-slave-j0qf krb5kdc[15464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:36:10 dist-test-slave-j0qf krb5kdc[15464](info): set up 2 sockets
Sep 28 06:36:10 dist-test-slave-j0qf krb5kdc[15464](info): commencing operation
krb5kdc: starting...
W20260928 06:36:11.221928 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:36:11 dist-test-slave-j0qf krb5kdc[15464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:36:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:36:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:36:11.349377 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:43253
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:46551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:43253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:11.459823 15477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:11.460114 15477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:11.460183 15477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:11.464107 15477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:36:11.464188 15477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:11.464227 15477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:11.464282 15477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:36:11.464321 15477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:36:11.469411 15477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:46551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:43253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:43253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:36:11.470435 15477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:11.471436 15477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:11.479712 15482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:11.479789 15477 server_base.cc:1034] running on GCE node
W20260928 06:36:11.479770 15483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:11.479948 15485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:11.480504 15477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:11.481575 15477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:11.482738 15477 hybrid_clock.cc:648] HybridClock initialized: now 1790577371482711 us; error 41 us; skew 500 ppm
Sep 28 06:36:11 dist-test-slave-j0qf krb5kdc[15464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577371, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:11.492141 15477 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:36:11.493165 15477 webserver.cc:460] Webserver started at http://127.5.22.62:35197/ using document root <none> and password file <none>
I20260928 06:36:11.493743 15477 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:11.493815 15477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:11.494055 15477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:11.496248 15477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "bc0dc00f76c54606903c71884aae6f1c"
format_stamp: "Formatted at 2026-09-28 06:36:11 on dist-test-slave-j0qf"
server_key: "3ea59539618b4d1ec16eac5f27a3f56b"
server_key_iv: "36cb1c0b0a266caf9d07a6c872b99790"
server_key_version: "encryptionkey@0"
I20260928 06:36:11.496819 15477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "bc0dc00f76c54606903c71884aae6f1c"
format_stamp: "Formatted at 2026-09-28 06:36:11 on dist-test-slave-j0qf"
server_key: "3ea59539618b4d1ec16eac5f27a3f56b"
server_key_iv: "36cb1c0b0a266caf9d07a6c872b99790"
server_key_version: "encryptionkey@0"
I20260928 06:36:11.500595 15477 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 06:36:11.503113 15492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:11.504400 15477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:36:11.504575 15477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "bc0dc00f76c54606903c71884aae6f1c"
format_stamp: "Formatted at 2026-09-28 06:36:11 on dist-test-slave-j0qf"
server_key: "3ea59539618b4d1ec16eac5f27a3f56b"
server_key_iv: "36cb1c0b0a266caf9d07a6c872b99790"
server_key_version: "encryptionkey@0"
I20260928 06:36:11.504683 15477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:11.527004 15477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:11.530388 15477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:11.530602 15477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:11.537734 15477 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:43253
I20260928 06:36:11.538017 15544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:43253 every 8 connection(s)
I20260928 06:36:11.538781 15477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:36:11.541637 15545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:11.545977 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 15477
I20260928 06:36:11.546099 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:36:11.546420 5208 external_mini_cluster.cc:1428] Setting key 148fbf134ba16734eb4486750d89df41
I20260928 06:36:11.547689 15545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc0dc00f76c54606903c71884aae6f1c: Bootstrap starting.
I20260928 06:36:11.550031 15545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc0dc00f76c54606903c71884aae6f1c: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:11.550894 15545 log.cc:824] T 00000000000000000000000000000000 P bc0dc00f76c54606903c71884aae6f1c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:11.553056 15545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc0dc00f76c54606903c71884aae6f1c: No bootstrap required, opened a new log
Sep 28 06:36:11 dist-test-slave-j0qf krb5kdc[15464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:36:11.557327 15545 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc0dc00f76c54606903c71884aae6f1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0dc00f76c54606903c71884aae6f1c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43253 } }
I20260928 06:36:11.557745 15545 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc0dc00f76c54606903c71884aae6f1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:11.557824 15545 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc0dc00f76c54606903c71884aae6f1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc0dc00f76c54606903c71884aae6f1c, State: Initialized, Role: FOLLOWER
I20260928 06:36:11.558251 15545 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc0dc00f76c54606903c71884aae6f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0dc00f76c54606903c71884aae6f1c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43253 } }
I20260928 06:36:11.558446 15545 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc0dc00f76c54606903c71884aae6f1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:36:11.558516 15545 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc0dc00f76c54606903c71884aae6f1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:36:11.558611 15545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc0dc00f76c54606903c71884aae6f1c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:11.559482 15545 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc0dc00f76c54606903c71884aae6f1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0dc00f76c54606903c71884aae6f1c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43253 } }
I20260928 06:36:11.559830 15545 leader_election.cc:302] T 00000000000000000000000000000000 P bc0dc00f76c54606903c71884aae6f1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc0dc00f76c54606903c71884aae6f1c; no voters:
I20260928 06:36:11.560189 15545 leader_election.cc:288] T 00000000000000000000000000000000 P bc0dc00f76c54606903c71884aae6f1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:36:11.560259 15550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc0dc00f76c54606903c71884aae6f1c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:36:11.560535 15550 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc0dc00f76c54606903c71884aae6f1c [term 1 LEADER]: Becoming Leader. State: Replica: bc0dc00f76c54606903c71884aae6f1c, State: Running, Role: LEADER
I20260928 06:36:11.560880 15550 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc0dc00f76c54606903c71884aae6f1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0dc00f76c54606903c71884aae6f1c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43253 } }
I20260928 06:36:11.561455 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:11.547901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46194 (local address 127.5.22.62:43253)
0928 06:36:11.548335 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:11.548361 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:11.548396 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:36:11.549010 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:11.549027 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:11.549337 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:11.549618 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:11.550594 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:11.551632 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:11.552807 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:11.553370 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:11.553514 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:11.556310 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:11.556315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:11.556328 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:11.556354 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:11.558286 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:11.558862 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:11.558868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:11.558874 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:11.559006 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:11.559301 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:11.559305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:11.559307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:11.559751 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:11.559957 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:11.560248 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 06:36:11.560567 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":139,"threads_started":1}
I20260928 06:36:11.561564 15545 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc0dc00f76c54606903c71884aae6f1c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:36:11.562501 15551 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc0dc00f76c54606903c71884aae6f1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc0dc00f76c54606903c71884aae6f1c. Latest consensus state: current_term: 1 leader_uuid: "bc0dc00f76c54606903c71884aae6f1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0dc00f76c54606903c71884aae6f1c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43253 } } }
I20260928 06:36:11.562644 15551 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc0dc00f76c54606903c71884aae6f1c [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:11.562846 15551 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc0dc00f76c54606903c71884aae6f1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc0dc00f76c54606903c71884aae6f1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0dc00f76c54606903c71884aae6f1c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43253 } } }
I20260928 06:36:11.562952 15551 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc0dc00f76c54606903c71884aae6f1c [sys.catalog]: This master's current role is: LEADER
W20260928 06:36:11.566421 15566 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc0dc00f76c54606903c71884aae6f1c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:36:11.566506 15566 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:36:11.566573 15557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:36:11.567235 15557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:36:11.572237 15557 catalog_manager.cc:1347] Generated new cluster ID: dc57f76377aa4f0185e5d29e43513ffb
I20260928 06:36:11.572302 15557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:36:11.588335 15557 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:36:11.589663 15557 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:36:11.597878 15557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc0dc00f76c54606903c71884aae6f1c: Generated new TSK 0
I20260928 06:36:11.598732 15557 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.22.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
I20260928 06:36:11.633347 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:43253
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:46551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:11.743487 15571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:11.743743 15571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:11.743784 15571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:11.747745 15571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:11.747807 15571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:11.747864 15571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:36:11.753134 15571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:46551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:43253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:36:11.754208 15571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:11.755303 15571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:11.765409 15579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:11.765499 15576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:11.765424 15577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:11.765775 15571 server_base.cc:1034] running on GCE node
I20260928 06:36:11.766161 15571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:11.766816 15571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:11.767976 15571 hybrid_clock.cc:648] HybridClock initialized: now 1790577371767963 us; error 35 us; skew 500 ppm
Sep 28 06:36:11 dist-test-slave-j0qf krb5kdc[15464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577371, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:11.770788 15571 init.cc:377] Logged in from keytab as kudu/127.5.22.1@KRBTEST.COM (short username kudu)
I20260928 06:36:11.771625 15571 webserver.cc:460] Webserver started at http://127.5.22.1:42039/ using document root <none> and password file <none>
I20260928 06:36:11.772135 15571 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:11.772240 15571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:11.772419 15571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:11.774426 15571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "939b39c03d454b299644861abfce9757"
format_stamp: "Formatted at 2026-09-28 06:36:11 on dist-test-slave-j0qf"
server_key: "96b5779a0acd3ef334c423f1af69c6d8"
server_key_iv: "89ba414450011e95b829c7218bd63099"
server_key_version: "encryptionkey@0"
I20260928 06:36:11.774947 15571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "939b39c03d454b299644861abfce9757"
format_stamp: "Formatted at 2026-09-28 06:36:11 on dist-test-slave-j0qf"
server_key: "96b5779a0acd3ef334c423f1af69c6d8"
server_key_iv: "89ba414450011e95b829c7218bd63099"
server_key_version: "encryptionkey@0"
I20260928 06:36:11.778550 15571 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260928 06:36:11.780896 15586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:11.782155 15571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:36:11.782286 15571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "939b39c03d454b299644861abfce9757"
format_stamp: "Formatted at 2026-09-28 06:36:11 on dist-test-slave-j0qf"
server_key: "96b5779a0acd3ef334c423f1af69c6d8"
server_key_iv: "89ba414450011e95b829c7218bd63099"
server_key_version: "encryptionkey@0"
I20260928 06:36:11.782450 15571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:11.793952 15571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:11.797187 15571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:11.797389 15571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:11.797950 15571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:11.798799 15571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:11.798870 15571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:11.798934 15571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:11.798986 15571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:11.807694 15571 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:42025
I20260928 06:36:11.807708 15699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:42025 every 8 connection(s)
I20260928 06:36:11.808686 15571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:36:11.809589 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 15571
I20260928 06:36:11.809741 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:36:11.810031 5208 external_mini_cluster.cc:1428] Setting key bc9f5db020e714d91eee09db8543ecf2
WARNING: no policy specified for kudu/127.5.22.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.2@KRBTEST.COM" created.
Sep 28 06:36:11 dist-test-slave-j0qf krb5kdc[15464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577371, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:36:11.827570 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:11.811577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:42715 (local address 127.5.22.62:43253)
0928 06:36:11.811698 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:11.811702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:11.812793 (+ 1091us) server_negotiation.cc:407] Connection header received
0928 06:36:11.814312 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:11.814335 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:11.814398 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:11.814489 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:11.815730 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:11.816671 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:11.817822 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:11.818210 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:11.818380 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:11.823502 (+ 5122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:11.823506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:11.823509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:11.823544 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:11.825060 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:11.825750 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:11.825753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:11.825754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:11.825833 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:11.826246 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:11.826248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:11.826249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:11.826544 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:11.826645 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:11.827322 (+ 677us) server_negotiation.cc:299] Negotiation successful
0928 06:36:11.827431 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:36:11.828950 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:11.812051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43253 (local address 127.5.22.1:42715)
0928 06:36:11.812601 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:11.812661 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:11.814098 (+ 1437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:11.814657 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:11.814663 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:11.815013 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:11.815539 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:11.815561 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:11.816812 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:11.816815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:11.817694 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:11.817702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:11.818421 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:11.818424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:11.818565 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:11.819206 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:11.819223 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:11.823306 (+ 4083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:11.825210 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:11.825219 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:11.825240 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:11.825644 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:11.825927 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:11.825933 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:11.825938 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:11.826145 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:11.826670 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:11.826678 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:11.827039 (+ 361us) client_negotiation.cc:769] Sending connection context
0928 06:36:11.827292 (+ 253us) client_negotiation.cc:240] Negotiation successful
0928 06:36:11.827717 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":119,"threads_started":1}
I20260928 06:36:11.830363 15700 heartbeater.cc:347] Connected to a master server at 127.5.22.62:43253
I20260928 06:36:11.830623 15700 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:11.831133 15700 heartbeater.cc:511] Master 127.5.22.62:43253 requested a full tablet report, sending...
I20260928 06:36:11.832736 15509 ts_manager.cc:194] Registered new tserver with Master: 939b39c03d454b299644861abfce9757 (127.5.22.1:42025)
I20260928 06:36:11.833920 15509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.1@KRBTEST.COM'} at 127.5.22.1:42715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
I20260928 06:36:11.839733 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:43253
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:46551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:11.953159 15705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:11.953505 15705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:11.953596 15705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:11.957727 15705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:11.957834 15705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:11.957933 15705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:36:11.963799 15705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:46551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:43253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:36:11.964975 15705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:11.966095 15705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:11.976212 15711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:11.976266 15705 server_base.cc:1034] running on GCE node
W20260928 06:36:11.976466 15710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:11.976235 15713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:11.976994 15705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:11.977645 15705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:11.978804 15705 hybrid_clock.cc:648] HybridClock initialized: now 1790577371978790 us; error 27 us; skew 500 ppm
Sep 28 06:36:11 dist-test-slave-j0qf krb5kdc[15464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577371, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:11.981585 15705 init.cc:377] Logged in from keytab as kudu/127.5.22.2@KRBTEST.COM (short username kudu)
I20260928 06:36:11.982565 15705 webserver.cc:460] Webserver started at http://127.5.22.2:40131/ using document root <none> and password file <none>
I20260928 06:36:11.983134 15705 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:11.983227 15705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:11.983461 15705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:11.985544 15705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "8a74c3c4889a4d1092158e54de8f414d"
format_stamp: "Formatted at 2026-09-28 06:36:11 on dist-test-slave-j0qf"
server_key: "6e53f1005bb0364a9fe35b4573f47427"
server_key_iv: "304b0b50c86211a59192e24d1575fa5f"
server_key_version: "encryptionkey@0"
I20260928 06:36:11.986177 15705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "8a74c3c4889a4d1092158e54de8f414d"
format_stamp: "Formatted at 2026-09-28 06:36:11 on dist-test-slave-j0qf"
server_key: "6e53f1005bb0364a9fe35b4573f47427"
server_key_iv: "304b0b50c86211a59192e24d1575fa5f"
server_key_version: "encryptionkey@0"
I20260928 06:36:11.989759 15705 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:36:11.992097 15720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:11.993316 15705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:36:11.993451 15705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "8a74c3c4889a4d1092158e54de8f414d"
format_stamp: "Formatted at 2026-09-28 06:36:11 on dist-test-slave-j0qf"
server_key: "6e53f1005bb0364a9fe35b4573f47427"
server_key_iv: "304b0b50c86211a59192e24d1575fa5f"
server_key_version: "encryptionkey@0"
I20260928 06:36:11.993558 15705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:12.028772 15705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:12.032315 15705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:12.032555 15705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:12.033195 15705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:12.034078 15705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:12.034152 15705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:12.034231 15705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:12.034276 15705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:12.043627 15705 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:41867
I20260928 06:36:12.043680 15833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:41867 every 8 connection(s)
I20260928 06:36:12.044538 15705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:36:12.047152 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 15705
I20260928 06:36:12.047248 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:36:12.047632 5208 external_mini_cluster.cc:1428] Setting key 4479db2a719a1c60b5c9716f59de5e0d
Sep 28 06:36:12 dist-test-slave-j0qf krb5kdc[15464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577371, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.22.3@KRBTEST.COM; defaulting to no policy
I20260928 06:36:12.061105 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:12.046568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:44083 (local address 127.5.22.62:43253)
0928 06:36:12.046736 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:12.046740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:12.047748 (+ 1008us) server_negotiation.cc:407] Connection header received
0928 06:36:12.049388 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:12.049392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:12.049449 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:12.049546 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:12.051188 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.051803 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.053347 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.053760 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.053900 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:12.056800 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:12.056806 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:12.056811 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:12.056845 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:12.058719 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:12.059299 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:12.059301 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:12.059303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:12.059377 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:12.059774 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:12.059777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:12.059778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:12.060013 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:12.060093 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:12.060758 (+ 665us) server_negotiation.cc:299] Negotiation successful
0928 06:36:12.060908 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.3@KRBTEST.COM" created.
I20260928 06:36:12.062170 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:12.047002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43253 (local address 127.5.22.2:44083)
0928 06:36:12.047576 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:12.047637 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:12.049124 (+ 1487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:12.049741 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:12.049750 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:12.050289 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:12.051025 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:12.051047 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.051957 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.051964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:12.053207 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:12.053216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.053911 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.053913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:12.054090 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:12.054793 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:12.054820 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:12.056628 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:12.058891 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:12.058899 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:12.058916 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:12.059183 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:12.059509 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:12.059519 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:12.059521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:12.059675 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:12.060147 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:12.060158 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:12.060488 (+ 330us) client_negotiation.cc:769] Sending connection context
0928 06:36:12.060659 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 06:36:12.061023 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":153,"threads_started":1}
I20260928 06:36:12.064055 15834 heartbeater.cc:347] Connected to a master server at 127.5.22.62:43253
I20260928 06:36:12.064350 15834 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:12.065044 15834 heartbeater.cc:511] Master 127.5.22.62:43253 requested a full tablet report, sending...
I20260928 06:36:12.066214 15509 ts_manager.cc:194] Registered new tserver with Master: 8a74c3c4889a4d1092158e54de8f414d (127.5.22.2:41867)
I20260928 06:36:12.066908 15509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.2@KRBTEST.COM'} at 127.5.22.2:44083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
I20260928 06:36:12.078140 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:43253
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:46551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:12.187842 15839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:12.188110 15839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:12.188154 15839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:12.192229 15839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:12.192292 15839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:12.192369 15839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:36:12.197988 15839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:46551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:43253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:36:12.199173 15839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:12.200255 15839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:12.210235 15844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:12.210237 15845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:12.210237 15847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:12.210409 15839 server_base.cc:1034] running on GCE node
I20260928 06:36:12.211004 15839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:12.211634 15839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:12.212850 15839 hybrid_clock.cc:648] HybridClock initialized: now 1790577372212836 us; error 26 us; skew 500 ppm
Sep 28 06:36:12 dist-test-slave-j0qf krb5kdc[15464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577372, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:12.215629 15839 init.cc:377] Logged in from keytab as kudu/127.5.22.3@KRBTEST.COM (short username kudu)
I20260928 06:36:12.216490 15839 webserver.cc:460] Webserver started at http://127.5.22.3:38721/ using document root <none> and password file <none>
I20260928 06:36:12.217063 15839 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:12.217155 15839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:12.217389 15839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:12.219503 15839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "b70d6372dbe649a4bb20c62700ccd023"
format_stamp: "Formatted at 2026-09-28 06:36:12 on dist-test-slave-j0qf"
server_key: "7ba2327b9703177d649e8e33e2d1f1f1"
server_key_iv: "3bb6e1cebe6e1f98689804b0388957d2"
server_key_version: "encryptionkey@0"
I20260928 06:36:12.220086 15839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "b70d6372dbe649a4bb20c62700ccd023"
format_stamp: "Formatted at 2026-09-28 06:36:12 on dist-test-slave-j0qf"
server_key: "7ba2327b9703177d649e8e33e2d1f1f1"
server_key_iv: "3bb6e1cebe6e1f98689804b0388957d2"
server_key_version: "encryptionkey@0"
I20260928 06:36:12.223873 15839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 06:36:12.226219 15854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:12.227540 15839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:36:12.227671 15839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "b70d6372dbe649a4bb20c62700ccd023"
format_stamp: "Formatted at 2026-09-28 06:36:12 on dist-test-slave-j0qf"
server_key: "7ba2327b9703177d649e8e33e2d1f1f1"
server_key_iv: "3bb6e1cebe6e1f98689804b0388957d2"
server_key_version: "encryptionkey@0"
I20260928 06:36:12.227779 15839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:12.263803 15839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:12.266848 15839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:12.267061 15839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:12.267655 15839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:12.268486 15839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:12.268554 15839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:12.268630 15839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:12.268674 15839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:12.277278 15839 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:46379
I20260928 06:36:12.277348 15967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:46379 every 8 connection(s)
I20260928 06:36:12.278206 15839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:36:12.284982 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 15839
I20260928 06:36:12.285144 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
I20260928 06:36:12.285454 5208 external_mini_cluster.cc:1428] Setting key 51881851bd293d574eb4a419c8fbdbdb
Sep 28 06:36:12 dist-test-slave-j0qf krb5kdc[15464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577372, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:36:12.292026 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:12.280126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:45693 (local address 127.5.22.62:43253)
0928 06:36:12.280266 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:12.280269 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:12.281047 (+ 778us) server_negotiation.cc:407] Connection header received
0928 06:36:12.282433 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:12.282436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:12.282492 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:12.282600 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:12.283727 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.284263 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.285627 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.285901 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.286059 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:12.288641 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:12.288645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:12.288648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:12.288674 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:12.289928 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:12.290453 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:12.290457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:12.290459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:12.290577 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:12.291053 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:12.291056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:12.291057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:12.291240 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:12.291324 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:12.291811 (+ 487us) server_negotiation.cc:299] Negotiation successful
0928 06:36:12.291911 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:36:12.292897 15970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:12.280460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43253 (local address 127.5.22.3:45693)
0928 06:36:12.280899 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:12.280941 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:12.282179 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:12.282768 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:12.282775 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:12.283114 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:12.283579 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:12.283591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.284395 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.284403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:12.285453 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:12.285462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.286160 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.286165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:12.286383 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:12.287038 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:12.287054 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:12.288479 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:12.290056 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:12.290061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:12.290075 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:12.290368 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:12.290680 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:12.290684 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:12.290686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:12.290841 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:12.291337 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:12.291342 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:12.291597 (+ 255us) client_negotiation.cc:769] Sending connection context
0928 06:36:12.291745 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 06:36:12.292007 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":109,"threads_started":1}
I20260928 06:36:12.294044 15968 heartbeater.cc:347] Connected to a master server at 127.5.22.62:43253
I20260928 06:36:12.294306 15968 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:12.294821 15968 heartbeater.cc:511] Master 127.5.22.62:43253 requested a full tablet report, sending...
I20260928 06:36:12.295976 15509 ts_manager.cc:194] Registered new tserver with Master: b70d6372dbe649a4bb20c62700ccd023 (127.5.22.3:46379)
I20260928 06:36:12.296650 15509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.3@KRBTEST.COM'} at 127.5.22.3:45693
I20260928 06:36:12.299512 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:36:12.309079 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:12.301617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46196 (local address 127.5.22.62:43253)
0928 06:36:12.301738 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:12.301745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:12.301877 (+ 132us) server_negotiation.cc:407] Connection header received
0928 06:36:12.301986 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:12.301988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:12.302024 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:12.302094 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:12.302919 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.303425 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.304466 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.304665 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.304739 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:12.306026 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:12.306029 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:12.306031 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:12.306052 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:12.307387 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:12.307855 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:12.307857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:12.307858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:12.307955 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:12.308200 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:12.308202 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:12.308203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:12.308395 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:12.308501 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:12.308729 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 06:36:12.308869 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:36:12.311828 15509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:36:12.314896 15509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:36:12.329480 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:12.323908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33580 (local address 127.5.22.2:41867)
0928 06:36:12.324306 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:12.324318 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:12.324344 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:36:12.324563 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:12.324569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:12.324788 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:12.324960 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:12.325515 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.326362 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.327917 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.328758 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.328868 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:12.329008 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:12.329206 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 06:36:12.329308 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":128,"threads_started":1}
I20260928 06:36:12.329468 15981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:12.323684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:41867 (local address 127.0.0.1:33580)
0928 06:36:12.324200 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:12.324214 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:12.324315 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:12.325196 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:12.325202 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:12.325230 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:12.325380 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:12.325389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.326496 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.326501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:12.327773 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:12.327784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.328891 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.328898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:12.329007 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:12.329043 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 06:36:12.329156 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 06:36:12.329303 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":21,"thread_start_us":115,"threads_started":1}
I20260928 06:36:12.329484 15979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:12.323486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:42025 (local address 127.0.0.1:35824)
0928 06:36:12.324129 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:12.324180 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:12.324311 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:12.325403 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:12.325406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:12.325428 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:12.325518 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:12.325525 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.326929 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.326933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:12.328134 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:12.328143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.329148 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.329153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:12.329254 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:12.329273 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 06:36:12.329336 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 06:36:12.329391 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":49,"thread_start_us":95,"threads_started":1}
I20260928 06:36:12.329785 15980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:12.323643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35824 (local address 127.5.22.1:42025)
0928 06:36:12.324966 (+ 1323us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:12.324982 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:12.325005 (+ 23us) server_negotiation.cc:407] Connection header received
0928 06:36:12.325082 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:12.325086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:12.325269 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:12.325424 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:12.325628 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.326789 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.328257 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.328838 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.328951 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:12.329084 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:12.329547 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 06:36:12.329648 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1225,"thread_start_us":82,"threads_started":1}
I20260928 06:36:12.329905 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:12.324245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:46379 (local address 127.0.0.1:57754)
0928 06:36:12.324654 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:12.324669 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:12.324748 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:12.325575 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:12.325580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:12.325601 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:12.325692 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:12.325697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.327140 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.327144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:12.328230 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:12.328241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.329304 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.329307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:12.329405 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:12.329428 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 06:36:12.329479 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:36:12.329561 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":31,"thread_start_us":85,"threads_started":1}
I20260928 06:36:12.330665 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:12.324626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57754 (local address 127.5.22.3:46379)
0928 06:36:12.325172 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:12.325183 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:12.325201 (+ 18us) server_negotiation.cc:407] Connection header received
0928 06:36:12.325264 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:12.325268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:12.325424 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:12.325830 (+ 406us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:12.325985 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.327016 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.328349 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.329145 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.330248 (+ 1103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:12.330410 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:12.330455 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 06:36:12.330534 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":139,"threads_started":1}
I20260928 06:36:12.332873 15634 tablet_service.cc:1467] Processing CreateTablet for tablet c975433fa7134819966b0714df9abf2d (DEFAULT_TABLE table=test-table [id=dd18a9cd0b2b4a04a23a633fd7022cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:12.333113 15902 tablet_service.cc:1467] Processing CreateTablet for tablet c975433fa7134819966b0714df9abf2d (DEFAULT_TABLE table=test-table [id=dd18a9cd0b2b4a04a23a633fd7022cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:12.333880 15634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c975433fa7134819966b0714df9abf2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:12.334024 15902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c975433fa7134819966b0714df9abf2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:12.336280 15768 tablet_service.cc:1467] Processing CreateTablet for tablet c975433fa7134819966b0714df9abf2d (DEFAULT_TABLE table=test-table [id=dd18a9cd0b2b4a04a23a633fd7022cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:12.337287 15768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c975433fa7134819966b0714df9abf2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:12.340185 15985 tablet_bootstrap.cc:492] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757: Bootstrap starting.
I20260928 06:36:12.341557 15986 tablet_bootstrap.cc:492] T c975433fa7134819966b0714df9abf2d P b70d6372dbe649a4bb20c62700ccd023: Bootstrap starting.
I20260928 06:36:12.342582 15985 tablet_bootstrap.cc:654] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:12.343446 15985 log.cc:824] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:12.343474 15987 tablet_bootstrap.cc:492] T c975433fa7134819966b0714df9abf2d P 8a74c3c4889a4d1092158e54de8f414d: Bootstrap starting.
I20260928 06:36:12.343694 15986 tablet_bootstrap.cc:654] T c975433fa7134819966b0714df9abf2d P b70d6372dbe649a4bb20c62700ccd023: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:12.345255 15986 log.cc:824] T c975433fa7134819966b0714df9abf2d P b70d6372dbe649a4bb20c62700ccd023: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:12.345521 15987 tablet_bootstrap.cc:654] T c975433fa7134819966b0714df9abf2d P 8a74c3c4889a4d1092158e54de8f414d: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:12.345723 15985 tablet_bootstrap.cc:492] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757: No bootstrap required, opened a new log
I20260928 06:36:12.345933 15985 ts_tablet_manager.cc:1397] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 06:36:12.346695 15987 log.cc:824] T c975433fa7134819966b0714df9abf2d P 8a74c3c4889a4d1092158e54de8f414d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:12.347371 15986 tablet_bootstrap.cc:492] T c975433fa7134819966b0714df9abf2d P b70d6372dbe649a4bb20c62700ccd023: No bootstrap required, opened a new log
I20260928 06:36:12.347604 15986 ts_tablet_manager.cc:1397] T c975433fa7134819966b0714df9abf2d P b70d6372dbe649a4bb20c62700ccd023: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 06:36:12.348307 15987 tablet_bootstrap.cc:492] T c975433fa7134819966b0714df9abf2d P 8a74c3c4889a4d1092158e54de8f414d: No bootstrap required, opened a new log
I20260928 06:36:12.348574 15987 ts_tablet_manager.cc:1397] T c975433fa7134819966b0714df9abf2d P 8a74c3c4889a4d1092158e54de8f414d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 06:36:12.350603 15985 raft_consensus.cc:348] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939b39c03d454b299644861abfce9757" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42025 } } peers { permanent_uuid: "8a74c3c4889a4d1092158e54de8f414d" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41867 } } peers { permanent_uuid: "b70d6372dbe649a4bb20c62700ccd023" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46379 } }
I20260928 06:36:12.350925 15985 raft_consensus.cc:374] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:12.351015 15985 raft_consensus.cc:729] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 939b39c03d454b299644861abfce9757, State: Initialized, Role: FOLLOWER
I20260928 06:36:12.351485 15985 consensus_queue.cc:260] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939b39c03d454b299644861abfce9757" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42025 } } peers { permanent_uuid: "8a74c3c4889a4d1092158e54de8f414d" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41867 } } peers { permanent_uuid: "b70d6372dbe649a4bb20c62700ccd023" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46379 } }
I20260928 06:36:12.351383 15986 raft_consensus.cc:348] T c975433fa7134819966b0714df9abf2d P b70d6372dbe649a4bb20c62700ccd023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939b39c03d454b299644861abfce9757" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42025 } } peers { permanent_uuid: "8a74c3c4889a4d1092158e54de8f414d" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41867 } } peers { permanent_uuid: "b70d6372dbe649a4bb20c62700ccd023" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46379 } }
I20260928 06:36:12.351377 15987 raft_consensus.cc:348] T c975433fa7134819966b0714df9abf2d P 8a74c3c4889a4d1092158e54de8f414d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939b39c03d454b299644861abfce9757" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42025 } } peers { permanent_uuid: "8a74c3c4889a4d1092158e54de8f414d" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41867 } } peers { permanent_uuid: "b70d6372dbe649a4bb20c62700ccd023" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46379 } }
I20260928 06:36:12.351625 15986 raft_consensus.cc:374] T c975433fa7134819966b0714df9abf2d P b70d6372dbe649a4bb20c62700ccd023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:12.351630 15987 raft_consensus.cc:374] T c975433fa7134819966b0714df9abf2d P 8a74c3c4889a4d1092158e54de8f414d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:12.351689 15987 raft_consensus.cc:729] T c975433fa7134819966b0714df9abf2d P 8a74c3c4889a4d1092158e54de8f414d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a74c3c4889a4d1092158e54de8f414d, State: Initialized, Role: FOLLOWER
I20260928 06:36:12.351699 15986 raft_consensus.cc:729] T c975433fa7134819966b0714df9abf2d P b70d6372dbe649a4bb20c62700ccd023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b70d6372dbe649a4bb20c62700ccd023, State: Initialized, Role: FOLLOWER
I20260928 06:36:12.352144 15986 consensus_queue.cc:260] T c975433fa7134819966b0714df9abf2d P b70d6372dbe649a4bb20c62700ccd023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939b39c03d454b299644861abfce9757" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42025 } } peers { permanent_uuid: "8a74c3c4889a4d1092158e54de8f414d" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41867 } } peers { permanent_uuid: "b70d6372dbe649a4bb20c62700ccd023" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46379 } }
I20260928 06:36:12.352213 15987 consensus_queue.cc:260] T c975433fa7134819966b0714df9abf2d P 8a74c3c4889a4d1092158e54de8f414d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939b39c03d454b299644861abfce9757" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42025 } } peers { permanent_uuid: "8a74c3c4889a4d1092158e54de8f414d" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41867 } } peers { permanent_uuid: "b70d6372dbe649a4bb20c62700ccd023" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46379 } }
I20260928 06:36:12.352699 15985 ts_tablet_manager.cc:1428] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 06:36:12.352855 15968 heartbeater.cc:503] Master 127.5.22.62:43253 was elected leader, sending a full tablet report...
I20260928 06:36:12.353109 15700 heartbeater.cc:503] Master 127.5.22.62:43253 was elected leader, sending a full tablet report...
I20260928 06:36:12.353127 15986 ts_tablet_manager.cc:1428] T c975433fa7134819966b0714df9abf2d P b70d6372dbe649a4bb20c62700ccd023: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 06:36:12.353168 15987 ts_tablet_manager.cc:1428] T c975433fa7134819966b0714df9abf2d P 8a74c3c4889a4d1092158e54de8f414d: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260928 06:36:12.353451 15834 heartbeater.cc:503] Master 127.5.22.62:43253 was elected leader, sending a full tablet report...
W20260928 06:36:12.529314 15969 tablet.cc:2369] T c975433fa7134819966b0714df9abf2d P b70d6372dbe649a4bb20c62700ccd023: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:36:12.545696 15835 tablet.cc:2369] T c975433fa7134819966b0714df9abf2d P 8a74c3c4889a4d1092158e54de8f414d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:36:12.560022 15701 tablet.cc:2369] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:36:12.656739 15991 raft_consensus.cc:482] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:36:12.657008 15991 raft_consensus.cc:504] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939b39c03d454b299644861abfce9757" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42025 } } peers { permanent_uuid: "8a74c3c4889a4d1092158e54de8f414d" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41867 } } peers { permanent_uuid: "b70d6372dbe649a4bb20c62700ccd023" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46379 } }
I20260928 06:36:12.658206 15991 leader_election.cc:288] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a74c3c4889a4d1092158e54de8f414d (127.5.22.2:41867), b70d6372dbe649a4bb20c62700ccd023 (127.5.22.3:46379)
I20260928 06:36:12.663902 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:12.659308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:44411 (local address 127.5.22.3:46379)
0928 06:36:12.659436 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:12.659442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:12.659600 (+ 158us) server_negotiation.cc:407] Connection header received
0928 06:36:12.659720 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:12.659722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:12.659762 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:12.659856 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:12.660139 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.660833 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.662472 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.663075 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.663198 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:12.663262 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:12.663696 (+ 434us) server_negotiation.cc:299] Negotiation successful
0928 06:36:12.663771 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:36:12.663902 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:12.658800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:32825 (local address 127.5.22.2:41867)
0928 06:36:12.658934 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:12.658938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:12.659264 (+ 326us) server_negotiation.cc:407] Connection header received
0928 06:36:12.659327 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:12.659333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:12.659386 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:12.659489 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:12.659857 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.660653 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.662472 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.663089 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.663203 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:12.663263 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:12.663696 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 06:36:12.663792 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 06:36:12.664045 15995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:12.659186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:46379 (local address 127.5.22.1:44411)
0928 06:36:12.659503 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:12.659517 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:12.659625 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:12.659907 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:12.659909 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:12.659924 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:12.660017 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:12.660021 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.660947 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.660950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:12.662266 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:12.662272 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.663217 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.663221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:12.663506 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:12.663573 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 06:36:12.663660 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:36:12.663803 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":265,"spinlock_wait_cycles":3712,"thread_start_us":96,"threads_started":1}
I20260928 06:36:12.664045 15994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:12.658734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:41867 (local address 127.5.22.1:32825)
0928 06:36:12.659061 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:12.659079 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:12.659201 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:12.659536 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:12.659542 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:12.659567 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:12.659735 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:12.659748 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.660769 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.660771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:12.662259 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:12.662271 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.663217 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.663221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:12.663506 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:12.663571 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 06:36:12.663658 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:36:12.663800 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":111,"spinlock_wait_cycles":13696,"thread_start_us":141,"threads_started":1}
I20260928 06:36:12.664798 15922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c975433fa7134819966b0714df9abf2d" candidate_uuid: "939b39c03d454b299644861abfce9757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b70d6372dbe649a4bb20c62700ccd023" is_pre_election: true
I20260928 06:36:12.664842 15788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c975433fa7134819966b0714df9abf2d" candidate_uuid: "939b39c03d454b299644861abfce9757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a74c3c4889a4d1092158e54de8f414d" is_pre_election: true
I20260928 06:36:12.665117 15922 raft_consensus.cc:2454] T c975433fa7134819966b0714df9abf2d P b70d6372dbe649a4bb20c62700ccd023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 939b39c03d454b299644861abfce9757 in term 0.
I20260928 06:36:12.665151 15788 raft_consensus.cc:2454] T c975433fa7134819966b0714df9abf2d P 8a74c3c4889a4d1092158e54de8f414d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 939b39c03d454b299644861abfce9757 in term 0.
I20260928 06:36:12.665649 15588 leader_election.cc:302] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a74c3c4889a4d1092158e54de8f414d, 939b39c03d454b299644861abfce9757; no voters:
I20260928 06:36:12.665949 15991 raft_consensus.cc:2781] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:36:12.666028 15991 raft_consensus.cc:482] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:36:12.666057 15991 raft_consensus.cc:3037] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:12.667173 15991 raft_consensus.cc:504] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939b39c03d454b299644861abfce9757" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42025 } } peers { permanent_uuid: "8a74c3c4889a4d1092158e54de8f414d" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41867 } } peers { permanent_uuid: "b70d6372dbe649a4bb20c62700ccd023" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46379 } }
I20260928 06:36:12.667519 15991 leader_election.cc:288] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757 [CANDIDATE]: Term 1 election: Requested vote from peers 8a74c3c4889a4d1092158e54de8f414d (127.5.22.2:41867), b70d6372dbe649a4bb20c62700ccd023 (127.5.22.3:46379)
I20260928 06:36:12.667986 15788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c975433fa7134819966b0714df9abf2d" candidate_uuid: "939b39c03d454b299644861abfce9757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a74c3c4889a4d1092158e54de8f414d"
I20260928 06:36:12.668047 15922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c975433fa7134819966b0714df9abf2d" candidate_uuid: "939b39c03d454b299644861abfce9757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b70d6372dbe649a4bb20c62700ccd023"
I20260928 06:36:12.668133 15788 raft_consensus.cc:3037] T c975433fa7134819966b0714df9abf2d P 8a74c3c4889a4d1092158e54de8f414d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:12.668176 15922 raft_consensus.cc:3037] T c975433fa7134819966b0714df9abf2d P b70d6372dbe649a4bb20c62700ccd023 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:12.669155 15788 raft_consensus.cc:2454] T c975433fa7134819966b0714df9abf2d P 8a74c3c4889a4d1092158e54de8f414d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 939b39c03d454b299644861abfce9757 in term 1.
I20260928 06:36:12.669159 15922 raft_consensus.cc:2454] T c975433fa7134819966b0714df9abf2d P b70d6372dbe649a4bb20c62700ccd023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 939b39c03d454b299644861abfce9757 in term 1.
I20260928 06:36:12.669466 15588 leader_election.cc:302] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 939b39c03d454b299644861abfce9757, b70d6372dbe649a4bb20c62700ccd023; no voters:
I20260928 06:36:12.669692 15991 raft_consensus.cc:2781] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:36:12.669950 15991 raft_consensus.cc:686] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757 [term 1 LEADER]: Becoming Leader. State: Replica: 939b39c03d454b299644861abfce9757, State: Running, Role: LEADER
I20260928 06:36:12.670285 15991 consensus_queue.cc:237] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939b39c03d454b299644861abfce9757" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42025 } } peers { permanent_uuid: "8a74c3c4889a4d1092158e54de8f414d" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41867 } } peers { permanent_uuid: "b70d6372dbe649a4bb20c62700ccd023" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46379 } }
I20260928 06:36:12.673615 15509 catalog_manager.cc:5495] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757 reported cstate change: term changed from 0 to 1, leader changed from <none> to 939b39c03d454b299644861abfce9757 (127.5.22.1). New cstate: current_term: 1 leader_uuid: "939b39c03d454b299644861abfce9757" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939b39c03d454b299644861abfce9757" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a74c3c4889a4d1092158e54de8f414d" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b70d6372dbe649a4bb20c62700ccd023" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46379 } health_report { overall_health: UNKNOWN } } }
I20260928 06:36:12.691380 15980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:12.687884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35828 (local address 127.5.22.1:42025)
0928 06:36:12.688017 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:12.688020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:12.688034 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:36:12.688114 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:12.688116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:12.688168 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:12.688257 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:36:12.688510 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.688995 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.689933 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.690151 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.690233 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:12.690571 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:36:12.690991 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:36:12.691095 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:12.691214 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 06:36:12.691268 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 06:36:12.697345 15922 raft_consensus.cc:1261] T c975433fa7134819966b0714df9abf2d P b70d6372dbe649a4bb20c62700ccd023 [term 1 FOLLOWER]: Refusing update from remote peer 939b39c03d454b299644861abfce9757: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:36:12.697357 15788 raft_consensus.cc:1261] T c975433fa7134819966b0714df9abf2d P 8a74c3c4889a4d1092158e54de8f414d [term 1 FOLLOWER]: Refusing update from remote peer 939b39c03d454b299644861abfce9757: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:36:12.698146 15996 consensus_queue.cc:1035] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a74c3c4889a4d1092158e54de8f414d" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:36:12.698343 15991 consensus_queue.cc:1035] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b70d6372dbe649a4bb20c62700ccd023" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:36:12.706058 16001 mvcc.cc:205] Tried to move back new op lower bound from 7334204918560788480 to 7334204918460293120. Current Snapshot: MvccSnapshot[applied={T|T < 7334204918560788480}]
I20260928 06:36:12.706615 16000 mvcc.cc:205] Tried to move back new op lower bound from 7334204918560788480 to 7334204918460293120. Current Snapshot: MvccSnapshot[applied={T|T < 7334204918560788480}]
I20260928 06:36:12.714778 15509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:36:12.714929 15509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:36:12.717324 15509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc0dc00f76c54606903c71884aae6f1c: Sending DeleteTablet for 3 replicas of tablet c975433fa7134819966b0714df9abf2d
I20260928 06:36:12.718114 15902 tablet_service.cc:1514] Processing DeleteTablet for tablet c975433fa7134819966b0714df9abf2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:12 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:57754
I20260928 06:36:12.718159 15634 tablet_service.cc:1514] Processing DeleteTablet for tablet c975433fa7134819966b0714df9abf2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:12 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:35824
I20260928 06:36:12.718161 15768 tablet_service.cc:1514] Processing DeleteTablet for tablet c975433fa7134819966b0714df9abf2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:12 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:33580
I20260928 06:36:12.718713 16009 tablet_replica.cc:330] T c975433fa7134819966b0714df9abf2d P b70d6372dbe649a4bb20c62700ccd023: stopping tablet replica
I20260928 06:36:12.718883 16010 tablet_replica.cc:330] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757: stopping tablet replica
I20260928 06:36:12.719079 16009 raft_consensus.cc:2229] T c975433fa7134819966b0714df9abf2d P b70d6372dbe649a4bb20c62700ccd023 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:36:12.719228 16010 raft_consensus.cc:2229] T c975433fa7134819966b0714df9abf2d P 939b39c03d454b299644861abfce9757 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:36:12.719334 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 15571
I20260928 06:36:12.719902 16009 raft_consensus.cc:2258] T c975433fa7134819966b0714df9abf2d P b70d6372dbe649a4bb20c62700ccd023 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:12.721592 16011 tablet_replica.cc:330] T c975433fa7134819966b0714df9abf2d P 8a74c3c4889a4d1092158e54de8f414d: stopping tablet replica
I20260928 06:36:12.721838 16009 ts_tablet_manager.cc:1905] T c975433fa7134819966b0714df9abf2d P b70d6372dbe649a4bb20c62700ccd023: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:12.721861 16011 raft_consensus.cc:2229] T c975433fa7134819966b0714df9abf2d P 8a74c3c4889a4d1092158e54de8f414d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:36:12.722146 16011 raft_consensus.cc:2258] T c975433fa7134819966b0714df9abf2d P 8a74c3c4889a4d1092158e54de8f414d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:12.723552 16011 ts_tablet_manager.cc:1905] T c975433fa7134819966b0714df9abf2d P 8a74c3c4889a4d1092158e54de8f414d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:12.724572 16009 ts_tablet_manager.cc:1918] T c975433fa7134819966b0714df9abf2d P b70d6372dbe649a4bb20c62700ccd023: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:36:12.724671 16009 log.cc:1196] T c975433fa7134819966b0714df9abf2d P b70d6372dbe649a4bb20c62700ccd023: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/c975433fa7134819966b0714df9abf2d
I20260928 06:36:12.725032 16009 ts_tablet_manager.cc:1939] T c975433fa7134819966b0714df9abf2d P b70d6372dbe649a4bb20c62700ccd023: Deleting consensus metadata
I20260928 06:36:12.726166 16011 ts_tablet_manager.cc:1918] T c975433fa7134819966b0714df9abf2d P 8a74c3c4889a4d1092158e54de8f414d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:36:12.726275 16011 log.cc:1196] T c975433fa7134819966b0714df9abf2d P 8a74c3c4889a4d1092158e54de8f414d: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577308481727-5208-0/minicluster-data/ts-1/wal/wals/c975433fa7134819966b0714df9abf2d
I20260928 06:36:12.726469 15494 catalog_manager.cc:4841] TS b70d6372dbe649a4bb20c62700ccd023 (127.5.22.3:46379): tablet c975433fa7134819966b0714df9abf2d (table test-table [id=dd18a9cd0b2b4a04a23a633fd7022cbe]) successfully deleted
I20260928 06:36:12.726620 16011 ts_tablet_manager.cc:1939] T c975433fa7134819966b0714df9abf2d P 8a74c3c4889a4d1092158e54de8f414d: Deleting consensus metadata
W20260928 06:36:12.726671 15495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.22.1:42025 (error 108)
I20260928 06:36:12.726919 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 15705
I20260928 06:36:12.728232 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:12.727885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:42025 (local address 127.0.0.1:35836)
0928 06:36:12.728042 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:12.728130 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.1:42025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 06:36:12.728395 15495 catalog_manager.cc:4584] TS 939b39c03d454b299644861abfce9757 (127.5.22.1:42025): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c975433fa7134819966b0714df9abf2d: Network error: Client connection negotiation failed: client connection to 127.5.22.1:42025: connect: Connection refused (error 111)
I20260928 06:36:12.732084 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 15839
I20260928 06:36:12.733048 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:12.732254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:41867 (local address 127.0.0.1:33596)
0928 06:36:12.732937 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:12.732984 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.2:41867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":613}
I20260928 06:36:12.737749 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 15477
2026-09-28T06:36:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2557 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:36:12 dist-test-slave-j0qf krb5kdc[16015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:36:12 dist-test-slave-j0qf krb5kdc[16015](info): set up 2 sockets
Sep 28 06:36:12 dist-test-slave-j0qf krb5kdc[16015](info): commencing operation
krb5kdc: starting...
W20260928 06:36:13.797863 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:36:13 dist-test-slave-j0qf krb5kdc[16015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:36:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:36:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.22.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.22.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.22.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/rangeradmin_127.5.22.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.22.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/rangeradmin_127.5.22.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.22.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.22.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.22.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/rangerlookup_127.5.22.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.22.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/rangerlookup_127.5.22.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.22.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.22.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.22.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/HTTP_127.5.22.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.22.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/HTTP_127.5.22.20@KRBTEST.COM.keytab.
I20260928 06:36:13.973861 5208 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskgGsXow/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/postgres -l logfile start
2026-09-28 06:36:16.881 UTC [16045] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 06:36:16.882 UTC [16045] LOG: listening on IPv4 address "127.5.22.20", port 43327
2026-09-28 06:36:16.885 UTC [16045] LOG: listening on Unix socket "/tmp/.s.PGSQL.43327"
2026-09-28 06:36:16.922 UTC [16048] LOG: database system was shut down at 2026-09-28 06:36:15 UTC
2026-09-28 06:36:16.926 UTC [16045] LOG: database system is ready to accept connections
I20260928 06:36:17.874473 5208 mini_postgres.cc:96] Postgres bound to 43327
2026-09-28 06:36:17.882 UTC [16056] FATAL: database "slave" does not exist
127.5.22.20:43327 - accepting connections
I20260928 06:36:17.883075 5208 mini_ranger.cc:161] Starting Ranger...
I20260928 06:36:17.901124 5208 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 06:36:18.441646 5208 mini_ranger.cc:87] Created ranger Postgres database
I20260928 06:36:18.441751 5208 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ranger-admin
2026-09-28 06:36:19,069 [I] DB FLAVOR :POSTGRES
2026-09-28 06:36:19,070 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:36:19,070 [I] Checking connection..
2026-09-28 06:36:19,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:43327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 06:36:19,357 [I] Checking connection passed.
2026-09-28 06:36:19,357 [I] --------- Verifying version history table ---------
2026-09-28 06:36:19,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:43327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:36:19,607 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:36:19,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:43327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:36:19,866 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:36:19,866 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 06:36:19,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:43327/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 06:36:20.104 UTC [16120] WARNING: there is no transaction in progress
2026-09-28 06:36:20,111 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 06:36:20,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:43327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:36:20,355 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 06:36:20,355 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:36:20,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:43327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 06:36:20,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:43327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 06:36:20,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:43327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j0qf.c.gcp-upstream.internal','N') ;"
2026-09-28 06:36:21,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:43327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:36:21,369 [I] Table x_portal_user does not exist in database ranger
2026-09-28 06:36:21,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:43327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:36:21,610 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 06:36:21,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:43327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:36:21,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:43327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:36:22,118 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 06:36:22,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:43327/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 06:36:23.147 UTC [16228] WARNING: there is no transaction in progress
2026-09-28 06:36:23.163 UTC [16228] WARNING: there is no transaction in progress
2026-09-28 06:36:23.179 UTC [16228] WARNING: there is no transaction in progress
2026-09-28 06:36:23.195 UTC [16228] WARNING: there is no transaction in progress
2026-09-28 06:36:23.213 UTC [16228] WARNING: there is no transaction in progress
2026-09-28 06:36:23.233 UTC [16228] WARNING: there is no transaction in progress
2026-09-28 06:36:23.301 UTC [16228] WARNING: there is no transaction in progress
2026-09-28 06:36:23.309 UTC [16228] WARNING: there is no transaction in progress
2026-09-28 06:36:23.324 UTC [16228] WARNING: there is no transaction in progress
2026-09-28 06:36:23.336 UTC [16228] WARNING: there is no transaction in progress
2026-09-28 06:36:23.347 UTC [16228] WARNING: there is no transaction in progress
2026-09-28 06:36:23.362 UTC [16228] WARNING: there is no transaction in progress
2026-09-28 06:36:23.373 UTC [16228] WARNING: there is no transaction in progress
2026-09-28 06:36:23.380 UTC [16228] WARNING: there is no transaction in progress
2026-09-28 06:36:23.387 UTC [16228] WARNING: there is no transaction in progress
2026-09-28 06:36:23.395 UTC [16228] WARNING: there is no transaction in progress
2026-09-28 06:36:24,086 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 06:36:24,086 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:43327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 06:36:24,328 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 06:36:24,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:43327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:36:24,554 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 06:36:24,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:43327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:36:24,809 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 06:36:24,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:43327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:36:25,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:43327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:36:25,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:43327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j0qf.c.gcp-upstream.internal';"
2026-09-28 06:36:25,557 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 06:36:25,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:43327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 06:36:25,798 [I] DB_PATCHES have already been applied
I20260928 06:36:25.803800 5208 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ranger-admin:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskgGsXow/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 06:36:25.803891 5208 mini_ranger.cc:193] Using host: 127.5.22.20
I20260928 06:36:25.807649 5208 mini_ranger.cc:227] Ranger admin URL: http://127.5.22.20:35563
Sep 28, 2026 6:36:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:36:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:36:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 6:36:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 6:36:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 6:36:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 6:36:27 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 06:36:28.060750 5208 mini_ranger.cc:160] Time spent starting Ranger: real 10.178s user 0.000s sys 0.013s
Sep 28, 2026 6:36:48 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 6:36:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 6:36:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 6:36:48 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 6:36:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 6:36:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 6:36:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 6:36:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 6:36:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 6:36:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 6:36:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 6:36:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 6:36:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 6:36:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 6:36:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 6:36:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 6:36:48 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 6:36:49 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 06:36:49.797190 5208 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.22.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.22.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.22.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/rangerkms_127.5.22.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.22.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/rangerkms_127.5.22.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.22.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.22.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.22.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/HTTP_127.5.22.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.22.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/HTTP_127.5.22.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 06:36:49 dist-test-slave-j0qf krb5kdc[16015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577409, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 06:36:49.891129 5208 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 06:36:49.904377 5208 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 06:36:50.847908 5208 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 06:36:50.848016 5208 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ranger-kms
I20260928 06:36:50.848132 5208 mini_ranger_kms.cc:227] Using postgres at 127.5.22.20:43327
2026-09-28 06:36:51,528 [I] DB FLAVOR :POSTGRES
2026-09-28 06:36:51,528 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:36:51,528 [I] Checking connection
2026-09-28 06:36:51,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:43327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 06:36:51,794 [I] connection success
2026-09-28 06:36:51,794 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 06:36:51,794 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 06:36:51,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:43327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 06:36:52,047 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 06:36:52,047 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:36:52,047 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 06:36:52,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:43327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 06:36:52,318 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 06:36:52.829736 5208 mini_ranger_kms.cc:310] Created kms service
I20260928 06:36:53.049149 5208 mini_ranger_kms.cc:326] Created kudu user
I20260928 06:36:53.101104 5208 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 06:36:53 dist-test-slave-j0qf krb5kdc[16015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577409, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.22.20@KRBTEST.COM
I20260928 06:36:53.519690 5208 mini_ranger_kms.cc:384] Added ranger policy
I20260928 06:36:53.519848 5208 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ranger-kms:/tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 06:36:53.519868 5208 mini_ranger_kms.cc:240] Using host: 127.5.22.20
I20260928 06:36:53.523802 5208 mini_ranger_kms.cc:276] Ranger KMS PID: 16410
I20260928 06:36:53.523941 5208 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.22.20:38811
Sep 28, 2026 6:36:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:36:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:36:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 6:36:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 6:36:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 6:36:55 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 06:36:55.923125 5208 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.032s user 0.038s sys 0.295s
I20260928 06:36:55.923278 5208 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 06:36:55.923412 5208 mini_ranger_kms.cc:401] 127.5.22.20:38811/kms/v1/keys
Sep 28 06:36:55 dist-test-slave-j0qf krb5kdc[16015](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577415, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.22.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 06:36:56 dist-test-slave-j0qf krb5kdc[16015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577415, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.22.20@KRBTEST.COM for HTTP/127.5.22.20@KRBTEST.COM
Sep 28 06:36:56 dist-test-slave-j0qf krb5kdc[16015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577415, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.22.20@KRBTEST.COM for HTTP/127.5.22.20@KRBTEST.COM
Sep 28 06:36:59 dist-test-slave-j0qf krb5kdc[16015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:36:59.122716 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:34085
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:35187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:34085
--ranger_config_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.22.20:38811/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:59.241320 16460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:59.241624 16460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:59.241667 16460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:59.245867 16460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:36:59.245941 16460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:59.245961 16460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:59.245977 16460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:36:59.245993 16460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:36:59.251883 16460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.22.20:38811/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:34085
--ranger_config_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:34085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:36:59.253034 16460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:59.254102 16460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:59.263202 16466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:59.263200 16465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:59.263574 16460 server_base.cc:1034] running on GCE node
W20260928 06:36:59.263718 16468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:59.264286 16460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:59.265482 16460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:59.266707 16460 hybrid_clock.cc:648] HybridClock initialized: now 1790577419266680 us; error 45 us; skew 500 ppm
Sep 28 06:36:59 dist-test-slave-j0qf krb5kdc[16015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577419, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:59.269906 16460 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:36:59.270949 16460 webserver.cc:460] Webserver started at http://127.5.22.62:32791/ using document root <none> and password file <none>
I20260928 06:36:59.271543 16460 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:59.271632 16460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:59.271845 16460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:36:59 dist-test-slave-j0qf krb5kdc[16015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577419, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for HTTP/127.5.22.20@KRBTEST.COM
I20260928 06:36:59.332188 16460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "2bc1b61539604a95aaaa9adcca273f8f"
format_stamp: "Formatted at 2026-09-28 06:36:59 on dist-test-slave-j0qf"
server_key: "ecf63e10f6c0a452b08745e39f2ad218"
server_key_iv: "54b71dcf062c419677e7e04be5872571"
server_key_version: "kuduclusterkey@0"
I20260928 06:36:59.332893 16460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "2bc1b61539604a95aaaa9adcca273f8f"
format_stamp: "Formatted at 2026-09-28 06:36:59 on dist-test-slave-j0qf"
server_key: "ecf63e10f6c0a452b08745e39f2ad218"
server_key_iv: "54b71dcf062c419677e7e04be5872571"
server_key_version: "kuduclusterkey@0"
I20260928 06:36:59.337103 16460 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 06:36:59.351984 16475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:59.353505 16460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 06:36:59.353662 16460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "2bc1b61539604a95aaaa9adcca273f8f"
format_stamp: "Formatted at 2026-09-28 06:36:59 on dist-test-slave-j0qf"
server_key: "ecf63e10f6c0a452b08745e39f2ad218"
server_key_iv: "54b71dcf062c419677e7e04be5872571"
server_key_version: "kuduclusterkey@0"
I20260928 06:36:59.353796 16460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:59.363955 16460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:59.367136 16460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:59.367637 16460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:59.376431 16460 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:34085
I20260928 06:36:59.376474 16527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:34085 every 8 connection(s)
I20260928 06:36:59.377759 16460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:36:59.380605 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 16460
I20260928 06:36:59.380741 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:36:59.380748 16528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:59.381076 5208 external_mini_cluster.cc:1428] Setting key c6dc143adcea8e789aad6fc9b500f832
I20260928 06:36:59.388020 16528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bc1b61539604a95aaaa9adcca273f8f: Bootstrap starting.
I20260928 06:36:59.390591 16528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bc1b61539604a95aaaa9adcca273f8f: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:59.391484 16528 log.cc:824] T 00000000000000000000000000000000 P 2bc1b61539604a95aaaa9adcca273f8f: Log is configured to *not* fsync() on all Append() calls
Sep 28 06:36:59 dist-test-slave-j0qf krb5kdc[16015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:36:59.393814 16528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bc1b61539604a95aaaa9adcca273f8f: No bootstrap required, opened a new log
I20260928 06:36:59.398125 16528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bc1b61539604a95aaaa9adcca273f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc1b61539604a95aaaa9adcca273f8f" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34085 } }
I20260928 06:36:59.398411 16528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bc1b61539604a95aaaa9adcca273f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:59.398509 16528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bc1b61539604a95aaaa9adcca273f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bc1b61539604a95aaaa9adcca273f8f, State: Initialized, Role: FOLLOWER
I20260928 06:36:59.398561 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:59.382564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54792 (local address 127.5.22.62:34085)
0928 06:36:59.383045 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:59.383064 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:59.383107 (+ 43us) server_negotiation.cc:407] Connection header received
0928 06:36:59.384258 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:59.384283 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:59.384718 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:59.385076 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:59.385965 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:59.387481 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:59.388641 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:59.389327 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:59.389490 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:59.392643 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:59.392649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:59.392665 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:59.392695 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:59.395375 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:59.395876 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:59.395882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:59.395888 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:59.396017 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:59.396317 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:59.396320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:59.396322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:59.396834 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:59.397020 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:59.397309 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 06:36:59.397596 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":147,"threads_started":1}
I20260928 06:36:59.399039 16528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bc1b61539604a95aaaa9adcca273f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc1b61539604a95aaaa9adcca273f8f" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34085 } }
I20260928 06:36:59.399192 16528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bc1b61539604a95aaaa9adcca273f8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:36:59.399272 16528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bc1b61539604a95aaaa9adcca273f8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:36:59.399380 16528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bc1b61539604a95aaaa9adcca273f8f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:59.400485 16528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bc1b61539604a95aaaa9adcca273f8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc1b61539604a95aaaa9adcca273f8f" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34085 } }
I20260928 06:36:59.400828 16528 leader_election.cc:302] T 00000000000000000000000000000000 P 2bc1b61539604a95aaaa9adcca273f8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bc1b61539604a95aaaa9adcca273f8f; no voters:
I20260928 06:36:59.401279 16528 leader_election.cc:288] T 00000000000000000000000000000000 P 2bc1b61539604a95aaaa9adcca273f8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:36:59.401350 16533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bc1b61539604a95aaaa9adcca273f8f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:36:59.401546 16533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bc1b61539604a95aaaa9adcca273f8f [term 1 LEADER]: Becoming Leader. State: Replica: 2bc1b61539604a95aaaa9adcca273f8f, State: Running, Role: LEADER
I20260928 06:36:59.401895 16533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bc1b61539604a95aaaa9adcca273f8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc1b61539604a95aaaa9adcca273f8f" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34085 } }
I20260928 06:36:59.402594 16528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bc1b61539604a95aaaa9adcca273f8f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:36:59.403880 16534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bc1b61539604a95aaaa9adcca273f8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bc1b61539604a95aaaa9adcca273f8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc1b61539604a95aaaa9adcca273f8f" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34085 } } }
I20260928 06:36:59.404000 16534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bc1b61539604a95aaaa9adcca273f8f [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:59.404872 16528 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 06:36:59.404408 16535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bc1b61539604a95aaaa9adcca273f8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bc1b61539604a95aaaa9adcca273f8f. Latest consensus state: current_term: 1 leader_uuid: "2bc1b61539604a95aaaa9adcca273f8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc1b61539604a95aaaa9adcca273f8f" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34085 } } }
I20260928 06:36:59.404969 16535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bc1b61539604a95aaaa9adcca273f8f [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:59.405357 16542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:36:59.411931 16542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:36:59.419823 16542 catalog_manager.cc:1347] Generated new cluster ID: 6dd30094938d4dd99963786cb9f2df02
I20260928 06:36:59.419894 16542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:36:59.435287 16542 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:36:59.437341 16542 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:36:59.445604 16542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bc1b61539604a95aaaa9adcca273f8f: Generated new TSK 0
I20260928 06:36:59.446628 16542 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.22.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
I20260928 06:37:00.163300 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.22.20:38811/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:34085
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:35187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:00.322831 16573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:00.323101 16573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:00.323144 16573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:00.327431 16573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:00.327488 16573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:00.327548 16573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:37:00.333077 16573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.22.20:38811/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:34085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:00.334225 16573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:00.335482 16573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:00.346997 16578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:00.347147 16581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:00.347656 16579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:00.348361 16573 server_base.cc:1034] running on GCE node
I20260928 06:37:00.348834 16573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:00.349535 16573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:00.350778 16573 hybrid_clock.cc:648] HybridClock initialized: now 1790577420350734 us; error 50 us; skew 500 ppm
Sep 28 06:37:00 dist-test-slave-j0qf krb5kdc[16015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577420, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:00.357290 16573 init.cc:377] Logged in from keytab as kudu/127.5.22.1@KRBTEST.COM (short username kudu)
I20260928 06:37:00.358250 16573 webserver.cc:460] Webserver started at http://127.5.22.1:41471/ using document root <none> and password file <none>
I20260928 06:37:00.358946 16573 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:00.359014 16573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:00.359253 16573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:37:00 dist-test-slave-j0qf krb5kdc[16015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577420, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for HTTP/127.5.22.20@KRBTEST.COM
I20260928 06:37:00.374261 16573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "71521cfa1c09413492f759488bcc529f"
format_stamp: "Formatted at 2026-09-28 06:37:00 on dist-test-slave-j0qf"
server_key: "7cdd4e20ea0baa0f16e37067518b6436"
server_key_iv: "0cae427b5508c2f43b5c8026c310268d"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:00.375152 16573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "71521cfa1c09413492f759488bcc529f"
format_stamp: "Formatted at 2026-09-28 06:37:00 on dist-test-slave-j0qf"
server_key: "7cdd4e20ea0baa0f16e37067518b6436"
server_key_iv: "0cae427b5508c2f43b5c8026c310268d"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:00.380656 16573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 06:37:00.401461 16590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:00.403355 16573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:37:00.403549 16573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "71521cfa1c09413492f759488bcc529f"
format_stamp: "Formatted at 2026-09-28 06:37:00 on dist-test-slave-j0qf"
server_key: "7cdd4e20ea0baa0f16e37067518b6436"
server_key_iv: "0cae427b5508c2f43b5c8026c310268d"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:00.403790 16573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:00.422925 16573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:00.426204 16573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:00.426440 16573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:00.427170 16573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:00.428193 16573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:00.428241 16573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:00.428293 16573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:00.428316 16573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:00.443622 16573 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:32931
I20260928 06:37:00.444810 16573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:37:00.445361 16703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:32931 every 8 connection(s)
I20260928 06:37:00.451086 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 16573
I20260928 06:37:00.451222 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:37:00.451538 5208 external_mini_cluster.cc:1428] Setting key 56f7640ac02180253cc95a4d7ba14e1c
Sep 28 06:37:00 dist-test-slave-j0qf krb5kdc[16015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577420, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.22.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.2@KRBTEST.COM" created.
I20260928 06:37:00.467278 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:00.448122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:53533 (local address 127.5.22.62:34085)
0928 06:37:00.453217 (+ 5095us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:00.453222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:00.453237 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:37:00.453544 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:00.453548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:00.453613 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:00.453696 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:00.455342 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:00.456225 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:00.458297 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:00.458691 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:00.458863 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:00.461958 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:00.461963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:00.461966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:00.461995 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:00.464393 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:00.465195 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:00.465199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:00.465201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:00.465318 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:00.465781 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:00.465786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:00.465788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:00.466037 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:00.466116 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:00.466923 (+ 807us) server_negotiation.cc:299] Negotiation successful
0928 06:37:00.467076 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5018,"thread_start_us":82,"threads_started":1}
I20260928 06:37:00.469357 16707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:00.448637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:34085 (local address 127.5.22.1:53533)
0928 06:37:00.450706 (+ 2069us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:00.450764 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:00.451837 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:00.454026 (+ 2189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:00.454038 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:00.454433 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:00.455162 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:00.455186 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:00.456376 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:00.456382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:00.458147 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:00.458162 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:00.458894 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:00.458899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:00.459118 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:00.459906 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:00.459928 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:00.461750 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:00.464531 (+ 2781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:00.464538 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:00.464558 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:00.465037 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:00.465453 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:00.465457 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:00.465460 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:00.465654 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:00.466209 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:00.466215 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:00.466607 (+ 392us) client_negotiation.cc:769] Sending connection context
0928 06:37:00.467476 (+ 869us) client_negotiation.cc:240] Negotiation successful
0928 06:37:00.468208 (+ 732us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1842,"thread_start_us":259,"threads_started":1}
I20260928 06:37:00.470822 16704 heartbeater.cc:347] Connected to a master server at 127.5.22.62:34085
I20260928 06:37:00.471093 16704 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:00.471613 16704 heartbeater.cc:511] Master 127.5.22.62:34085 requested a full tablet report, sending...
I20260928 06:37:00.473163 16492 ts_manager.cc:194] Registered new tserver with Master: 71521cfa1c09413492f759488bcc529f (127.5.22.1:32931)
I20260928 06:37:00.474288 16492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.1@KRBTEST.COM'} at 127.5.22.1:53533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
I20260928 06:37:00.486289 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.22.20:38811/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:34085
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:35187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:37:00 dist-test-slave-j0qf krb5kdc[16015](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577420, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 06:37:00.633734 16712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:00.634120 16712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:00.634199 16712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:00.638990 16712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:00.639066 16712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:00.639129 16712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:37:00.644807 16712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.22.20:38811/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:34085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:00.645874 16712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:00.647032 16712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:00.657323 16718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:00.657675 16721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:00.657833 16719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:00.658274 16712 server_base.cc:1034] running on GCE node
I20260928 06:37:00.658712 16712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:00.659363 16712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:00.660597 16712 hybrid_clock.cc:648] HybridClock initialized: now 1790577420660575 us; error 52 us; skew 500 ppm
Sep 28 06:37:00 dist-test-slave-j0qf krb5kdc[16015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577420, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:00.663851 16712 init.cc:377] Logged in from keytab as kudu/127.5.22.2@KRBTEST.COM (short username kudu)
I20260928 06:37:00.664978 16712 webserver.cc:460] Webserver started at http://127.5.22.2:33563/ using document root <none> and password file <none>
I20260928 06:37:00.665704 16712 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:00.665783 16712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:00.666023 16712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:37:00 dist-test-slave-j0qf krb5kdc[16015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577420, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for HTTP/127.5.22.20@KRBTEST.COM
I20260928 06:37:00.681880 16712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "b4c83f7193c54f6ebc73d132e83c77f4"
format_stamp: "Formatted at 2026-09-28 06:37:00 on dist-test-slave-j0qf"
server_key: "d9416ec1405e8d07ebf8fba43f7c9d64"
server_key_iv: "f0fa3176bc5f20e415068354b3a1789d"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:00.682699 16712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "b4c83f7193c54f6ebc73d132e83c77f4"
format_stamp: "Formatted at 2026-09-28 06:37:00 on dist-test-slave-j0qf"
server_key: "d9416ec1405e8d07ebf8fba43f7c9d64"
server_key_iv: "f0fa3176bc5f20e415068354b3a1789d"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:00.688017 16712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 06:37:00.710809 16728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:00.712571 16712 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260928 06:37:00.712723 16712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "b4c83f7193c54f6ebc73d132e83c77f4"
format_stamp: "Formatted at 2026-09-28 06:37:00 on dist-test-slave-j0qf"
server_key: "d9416ec1405e8d07ebf8fba43f7c9d64"
server_key_iv: "f0fa3176bc5f20e415068354b3a1789d"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:00.712868 16712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:00.729758 16712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:00.733682 16712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:00.733912 16712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:00.734717 16712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:00.735843 16712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:00.735913 16712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:00.735972 16712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:00.736006 16712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:00.747241 16712 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:35073
I20260928 06:37:00.748446 16712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:37:00.758033 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 16712
I20260928 06:37:00.758172 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:37:00.758486 5208 external_mini_cluster.cc:1428] Setting key f36b44eb6a74a72dc1d2d18e1556b74e
I20260928 06:37:00.758806 16841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:35073 every 8 connection(s)
WARNING: no policy specified for kudu/127.5.22.3@KRBTEST.COM; defaulting to no policy
Sep 28 06:37:00 dist-test-slave-j0qf krb5kdc[16015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577420, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.3@KRBTEST.COM" created.
I20260928 06:37:00.788426 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:00.763325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:33671 (local address 127.5.22.62:34085)
0928 06:37:00.763461 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:00.763465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:00.764064 (+ 599us) server_negotiation.cc:407] Connection header received
0928 06:37:00.766450 (+ 2386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:00.766454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:00.766512 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:00.766586 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:00.768944 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:00.769785 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:00.771437 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:00.771743 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:00.771835 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:00.783194 (+ 11359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:00.783200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:00.783203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:00.783238 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:00.785602 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:00.786355 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:00.786359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:00.786361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:00.786467 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:00.786909 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:00.786912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:00.786914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:00.787164 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:00.787268 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:00.788062 (+ 794us) server_negotiation.cc:299] Negotiation successful
0928 06:37:00.788221 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:37:00.790594 16845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:00.761999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:34085 (local address 127.5.22.2:33671)
0928 06:37:00.763905 (+ 1906us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:00.763964 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:00.765183 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:00.767072 (+ 1889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:00.767082 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:00.768045 (+ 963us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:00.768747 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:00.768768 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:00.769968 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:00.769971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:00.771295 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:00.771306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:00.772060 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:00.772067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:00.772324 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:00.778520 (+ 6196us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:00.778546 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:00.782925 (+ 4379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:00.785768 (+ 2843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:00.785775 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:00.785795 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:00.786184 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:00.786591 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:00.786594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:00.786596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:00.786789 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:00.787356 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:00.787363 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:00.787751 (+ 388us) client_negotiation.cc:769] Sending connection context
0928 06:37:00.789043 (+ 1292us) client_negotiation.cc:240] Negotiation successful
0928 06:37:00.789442 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1645,"thread_start_us":325,"threads_started":1}
I20260928 06:37:00.792821 16842 heartbeater.cc:347] Connected to a master server at 127.5.22.62:34085
I20260928 06:37:00.793092 16842 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:00.793721 16842 heartbeater.cc:511] Master 127.5.22.62:34085 requested a full tablet report, sending...
I20260928 06:37:00.794996 16492 ts_manager.cc:194] Registered new tserver with Master: b4c83f7193c54f6ebc73d132e83c77f4 (127.5.22.2:35073)
I20260928 06:37:00.795756 16492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.2@KRBTEST.COM'} at 127.5.22.2:33671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
I20260928 06:37:00.814729 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.22.20:38811/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:34085
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:35187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:00.990629 16847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:00.991035 16847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:00.991117 16847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:00.997663 16847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:00.997766 16847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:00.997861 16847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:37:01.006155 16847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.22.20:38811/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:34085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:01.007735 16847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:01.009186 16847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:01.023195 16853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:01.023629 16847 server_base.cc:1034] running on GCE node
W20260928 06:37:01.023793 16852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:01.024152 16855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:01.024719 16847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:01.025573 16847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:01.032992 16847 hybrid_clock.cc:648] HybridClock initialized: now 1790577421032896 us; error 100 us; skew 500 ppm
Sep 28 06:37:01 dist-test-slave-j0qf krb5kdc[16015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577421, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:01.036826 16847 init.cc:377] Logged in from keytab as kudu/127.5.22.3@KRBTEST.COM (short username kudu)
I20260928 06:37:01.038213 16847 webserver.cc:460] Webserver started at http://127.5.22.3:41573/ using document root <none> and password file <none>
I20260928 06:37:01.039022 16847 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:01.039127 16847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:01.039391 16847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:37:01 dist-test-slave-j0qf krb5kdc[16015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577421, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for HTTP/127.5.22.20@KRBTEST.COM
I20260928 06:37:01.073855 16847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "fe077003eb494bb78450f498da59abcd"
format_stamp: "Formatted at 2026-09-28 06:37:01 on dist-test-slave-j0qf"
server_key: "b92ba50df12ee1be4afdad5b34b8d6b7"
server_key_iv: "13b15c71c7e17a358a74e2812170c6c9"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:01.079138 16847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "fe077003eb494bb78450f498da59abcd"
format_stamp: "Formatted at 2026-09-28 06:37:01 on dist-test-slave-j0qf"
server_key: "b92ba50df12ee1be4afdad5b34b8d6b7"
server_key_iv: "13b15c71c7e17a358a74e2812170c6c9"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:01.097136 16847 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.005s sys 0.000s
Sep 28 06:37:01 dist-test-slave-j0qf krb5kdc[16015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577420, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for HTTP/127.5.22.20@KRBTEST.COM
I20260928 06:37:01.118813 16862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:01.120745 16847 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260928 06:37:01.120934 16847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "fe077003eb494bb78450f498da59abcd"
format_stamp: "Formatted at 2026-09-28 06:37:01 on dist-test-slave-j0qf"
server_key: "b92ba50df12ee1be4afdad5b34b8d6b7"
server_key_iv: "13b15c71c7e17a358a74e2812170c6c9"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:01.121096 16847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:01.175026 16847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:01.179631 16847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:01.179924 16847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:01.180675 16847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:01.181716 16847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:01.181806 16847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:01.181888 16847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:01.181939 16847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:01.194937 16847 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:43963
I20260928 06:37:01.196161 16847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:37:01.200676 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 16847
I20260928 06:37:01.200798 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
I20260928 06:37:01.201072 5208 external_mini_cluster.cc:1428] Setting key 93018f27db04cb9460d787711e92fc9d
I20260928 06:37:01.204919 16975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:43963 every 8 connection(s)
Sep 28 06:37:01 dist-test-slave-j0qf krb5kdc[16015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577421, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:37:01.220290 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:01.201462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:37507 (local address 127.5.22.62:34085)
0928 06:37:01.201837 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:01.201842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:01.206827 (+ 4985us) server_negotiation.cc:407] Connection header received
0928 06:37:01.208246 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:01.208249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:01.208301 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:01.208382 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:01.210103 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.210988 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.212718 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.213059 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.213160 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:01.216054 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:01.216058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:01.216061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:01.216088 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:01.217869 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:01.218523 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:01.218527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:01.218529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:01.218613 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:01.219004 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:01.219007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:01.219009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:01.219224 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:01.219302 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:01.219999 (+ 697us) server_negotiation.cc:299] Negotiation successful
0928 06:37:01.220128 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285}
I20260928 06:37:01.222124 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:01.199351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:34085 (local address 127.5.22.3:37507)
0928 06:37:01.206650 (+ 7299us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:01.206713 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:01.208004 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:01.208786 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:01.208796 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:01.209170 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:01.209866 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:01.209883 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.211124 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.211128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:01.212593 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:01.212603 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.213220 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.213223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:01.213426 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:01.214106 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:01.214125 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:01.215870 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:01.218014 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:01.218020 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:01.218038 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:01.218408 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:01.218723 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:01.218726 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:01.218728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:01.218898 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:01.219365 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:01.219386 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:01.219718 (+ 332us) client_negotiation.cc:769] Sending connection context
0928 06:37:01.220732 (+ 1014us) client_negotiation.cc:240] Negotiation successful
0928 06:37:01.221080 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7039,"thread_start_us":127,"threads_started":1}
I20260928 06:37:01.223591 16976 heartbeater.cc:347] Connected to a master server at 127.5.22.62:34085
I20260928 06:37:01.223855 16976 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:01.224428 16976 heartbeater.cc:511] Master 127.5.22.62:34085 requested a full tablet report, sending...
I20260928 06:37:01.225530 16492 ts_manager.cc:194] Registered new tserver with Master: fe077003eb494bb78450f498da59abcd (127.5.22.3:43963)
I20260928 06:37:01.226245 16492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.3@KRBTEST.COM'} at 127.5.22.3:37507
I20260928 06:37:01.229768 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:37:01.241549 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:01.232203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54794 (local address 127.5.22.62:34085)
0928 06:37:01.232327 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:01.232331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:01.232597 (+ 266us) server_negotiation.cc:407] Connection header received
0928 06:37:01.232697 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:01.232700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:01.232744 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:01.232824 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:01.233669 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.234492 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.236001 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.236339 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.236450 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:01.238073 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:01.238077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:01.238080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:01.238106 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:01.239848 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:01.240313 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:01.240316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:01.240318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:01.240410 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:01.240687 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:01.240690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:01.240691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:01.240898 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:01.241022 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:01.241268 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 06:37:01.241392 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:37:01.244899 16492 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:37:01.249274 16492 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:37:01.270849 16991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:01.260855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:32931 (local address 127.0.0.1:45726)
0928 06:37:01.261985 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:01.261999 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:01.262105 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:01.262828 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:01.262835 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:01.262858 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:01.262982 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:01.262989 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.264707 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.264711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:01.266191 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:01.266202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.270257 (+ 4055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.270261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:01.270414 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:01.270460 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 06:37:01.270571 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 06:37:01.270695 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"mutex_wait_us":3,"thread_start_us":91,"threads_started":1}
I20260928 06:37:01.273113 16992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:01.261323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39540 (local address 127.5.22.3:43963)
0928 06:37:01.266261 (+ 4938us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:01.266273 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:01.266298 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:37:01.266406 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:01.266412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:01.266617 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:01.266790 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:01.267740 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.268883 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.271323 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.272146 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.272247 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:01.272409 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:01.272873 (+ 464us) server_negotiation.cc:299] Negotiation successful
0928 06:37:01.272976 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4828,"thread_start_us":107,"threads_started":1}
I20260928 06:37:01.273524 16989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:01.260263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44796 (local address 127.5.22.2:35073)
0928 06:37:01.267077 (+ 6814us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:01.267093 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:01.267115 (+ 22us) server_negotiation.cc:407] Connection header received
0928 06:37:01.267263 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:01.267268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:01.267451 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:01.267893 (+ 442us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:01.268026 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.269908 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.271516 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.272358 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.273098 (+ 740us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:01.273238 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:01.273288 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 06:37:01.273387 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6709,"thread_start_us":127,"threads_started":1}
I20260928 06:37:01.272965 16990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:01.260133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:35073 (local address 127.0.0.1:44796)
0928 06:37:01.261246 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:01.261296 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:01.261438 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:01.267611 (+ 6173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:01.267614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:01.267632 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:01.267729 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:01.267735 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.270034 (+ 2299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.270038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:01.271337 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:01.271348 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.272504 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.272508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:01.272628 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:01.272654 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 06:37:01.272707 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 06:37:01.272777 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"mutex_wait_us":2,"thread_start_us":96,"threads_started":1}
I20260928 06:37:01.273572 16988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:01.261157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:43963 (local address 127.0.0.1:39540)
0928 06:37:01.263198 (+ 2041us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:01.263209 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:01.263278 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:01.267495 (+ 4217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:01.267498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:01.267517 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:01.267610 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:01.267615 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.269019 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.269023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:01.271171 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:01.271181 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.272471 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.272475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:01.272573 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:01.272595 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 06:37:01.272662 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:37:01.272727 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1983,"mutex_wait_us":928,"thread_start_us":72,"threads_started":1}
I20260928 06:37:01.277292 16910 tablet_service.cc:1467] Processing CreateTablet for tablet e25ef31f923c4c2c904aaa31a64c4d45 (DEFAULT_TABLE table=test-table [id=25be7d20a118442795af6b615baeb7b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:01.277520 16776 tablet_service.cc:1467] Processing CreateTablet for tablet e25ef31f923c4c2c904aaa31a64c4d45 (DEFAULT_TABLE table=test-table [id=25be7d20a118442795af6b615baeb7b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:01.278568 16776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25ef31f923c4c2c904aaa31a64c4d45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:01.278577 16910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25ef31f923c4c2c904aaa31a64c4d45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:01.287751 16993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:01.261532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45726 (local address 127.5.22.1:32931)
0928 06:37:01.261807 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:01.261814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:01.262200 (+ 386us) server_negotiation.cc:407] Connection header received
0928 06:37:01.262283 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:01.262288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:01.262533 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:01.262669 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:01.263391 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.264576 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.266350 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.267190 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.267290 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:01.267440 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:01.287367 (+ 19927us) server_negotiation.cc:299] Negotiation successful
0928 06:37:01.287531 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":71,"threads_started":1}
I20260928 06:37:01.290956 16638 tablet_service.cc:1467] Processing CreateTablet for tablet e25ef31f923c4c2c904aaa31a64c4d45 (DEFAULT_TABLE table=test-table [id=25be7d20a118442795af6b615baeb7b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:01.292127 16638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25ef31f923c4c2c904aaa31a64c4d45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:01.298723 16994 tablet_bootstrap.cc:492] T e25ef31f923c4c2c904aaa31a64c4d45 P fe077003eb494bb78450f498da59abcd: Bootstrap starting.
I20260928 06:37:01.301450 16994 tablet_bootstrap.cc:654] T e25ef31f923c4c2c904aaa31a64c4d45 P fe077003eb494bb78450f498da59abcd: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:01.302302 16994 log.cc:824] T e25ef31f923c4c2c904aaa31a64c4d45 P fe077003eb494bb78450f498da59abcd: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:01.304574 16994 tablet_bootstrap.cc:492] T e25ef31f923c4c2c904aaa31a64c4d45 P fe077003eb494bb78450f498da59abcd: No bootstrap required, opened a new log
I20260928 06:37:01.304807 16994 ts_tablet_manager.cc:1397] T e25ef31f923c4c2c904aaa31a64c4d45 P fe077003eb494bb78450f498da59abcd: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260928 06:37:01.309383 16994 raft_consensus.cc:348] T e25ef31f923c4c2c904aaa31a64c4d45 P fe077003eb494bb78450f498da59abcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71521cfa1c09413492f759488bcc529f" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 32931 } } peers { permanent_uuid: "fe077003eb494bb78450f498da59abcd" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43963 } } peers { permanent_uuid: "b4c83f7193c54f6ebc73d132e83c77f4" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35073 } }
I20260928 06:37:01.309666 16994 raft_consensus.cc:374] T e25ef31f923c4c2c904aaa31a64c4d45 P fe077003eb494bb78450f498da59abcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:01.309712 16994 raft_consensus.cc:729] T e25ef31f923c4c2c904aaa31a64c4d45 P fe077003eb494bb78450f498da59abcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe077003eb494bb78450f498da59abcd, State: Initialized, Role: FOLLOWER
I20260928 06:37:01.310460 16994 consensus_queue.cc:260] T e25ef31f923c4c2c904aaa31a64c4d45 P fe077003eb494bb78450f498da59abcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71521cfa1c09413492f759488bcc529f" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 32931 } } peers { permanent_uuid: "fe077003eb494bb78450f498da59abcd" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43963 } } peers { permanent_uuid: "b4c83f7193c54f6ebc73d132e83c77f4" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35073 } }
I20260928 06:37:01.311681 16994 ts_tablet_manager.cc:1428] T e25ef31f923c4c2c904aaa31a64c4d45 P fe077003eb494bb78450f498da59abcd: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 06:37:01.312134 16976 heartbeater.cc:503] Master 127.5.22.62:34085 was elected leader, sending a full tablet report...
I20260928 06:37:01.312433 16997 tablet_bootstrap.cc:492] T e25ef31f923c4c2c904aaa31a64c4d45 P b4c83f7193c54f6ebc73d132e83c77f4: Bootstrap starting.
I20260928 06:37:01.315014 16997 tablet_bootstrap.cc:654] T e25ef31f923c4c2c904aaa31a64c4d45 P b4c83f7193c54f6ebc73d132e83c77f4: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:01.316197 16997 log.cc:824] T e25ef31f923c4c2c904aaa31a64c4d45 P b4c83f7193c54f6ebc73d132e83c77f4: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:01.320097 16997 tablet_bootstrap.cc:492] T e25ef31f923c4c2c904aaa31a64c4d45 P b4c83f7193c54f6ebc73d132e83c77f4: No bootstrap required, opened a new log
I20260928 06:37:01.320395 16997 ts_tablet_manager.cc:1397] T e25ef31f923c4c2c904aaa31a64c4d45 P b4c83f7193c54f6ebc73d132e83c77f4: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 06:37:01.323489 16995 tablet_bootstrap.cc:492] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f: Bootstrap starting.
I20260928 06:37:01.325052 16997 raft_consensus.cc:348] T e25ef31f923c4c2c904aaa31a64c4d45 P b4c83f7193c54f6ebc73d132e83c77f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71521cfa1c09413492f759488bcc529f" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 32931 } } peers { permanent_uuid: "fe077003eb494bb78450f498da59abcd" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43963 } } peers { permanent_uuid: "b4c83f7193c54f6ebc73d132e83c77f4" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35073 } }
I20260928 06:37:01.325539 16997 raft_consensus.cc:374] T e25ef31f923c4c2c904aaa31a64c4d45 P b4c83f7193c54f6ebc73d132e83c77f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:01.325652 16997 raft_consensus.cc:729] T e25ef31f923c4c2c904aaa31a64c4d45 P b4c83f7193c54f6ebc73d132e83c77f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c83f7193c54f6ebc73d132e83c77f4, State: Initialized, Role: FOLLOWER
I20260928 06:37:01.326095 16995 tablet_bootstrap.cc:654] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:01.326272 16997 consensus_queue.cc:260] T e25ef31f923c4c2c904aaa31a64c4d45 P b4c83f7193c54f6ebc73d132e83c77f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71521cfa1c09413492f759488bcc529f" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 32931 } } peers { permanent_uuid: "fe077003eb494bb78450f498da59abcd" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43963 } } peers { permanent_uuid: "b4c83f7193c54f6ebc73d132e83c77f4" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35073 } }
I20260928 06:37:01.327059 16995 log.cc:824] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:01.327965 16997 ts_tablet_manager.cc:1428] T e25ef31f923c4c2c904aaa31a64c4d45 P b4c83f7193c54f6ebc73d132e83c77f4: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 06:37:01.328466 16842 heartbeater.cc:503] Master 127.5.22.62:34085 was elected leader, sending a full tablet report...
I20260928 06:37:01.335181 16995 tablet_bootstrap.cc:492] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f: No bootstrap required, opened a new log
I20260928 06:37:01.335536 16995 ts_tablet_manager.cc:1397] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20260928 06:37:01.340303 16995 raft_consensus.cc:348] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71521cfa1c09413492f759488bcc529f" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 32931 } } peers { permanent_uuid: "fe077003eb494bb78450f498da59abcd" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43963 } } peers { permanent_uuid: "b4c83f7193c54f6ebc73d132e83c77f4" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35073 } }
I20260928 06:37:01.340750 16995 raft_consensus.cc:374] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:01.340852 16995 raft_consensus.cc:729] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71521cfa1c09413492f759488bcc529f, State: Initialized, Role: FOLLOWER
I20260928 06:37:01.341572 16995 consensus_queue.cc:260] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71521cfa1c09413492f759488bcc529f" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 32931 } } peers { permanent_uuid: "fe077003eb494bb78450f498da59abcd" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43963 } } peers { permanent_uuid: "b4c83f7193c54f6ebc73d132e83c77f4" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35073 } }
I20260928 06:37:01.342844 16995 ts_tablet_manager.cc:1428] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 06:37:01.343487 16704 heartbeater.cc:503] Master 127.5.22.62:34085 was elected leader, sending a full tablet report...
I20260928 06:37:01.386788 17002 raft_consensus.cc:482] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:37:01.387065 17002 raft_consensus.cc:504] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71521cfa1c09413492f759488bcc529f" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 32931 } } peers { permanent_uuid: "fe077003eb494bb78450f498da59abcd" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43963 } } peers { permanent_uuid: "b4c83f7193c54f6ebc73d132e83c77f4" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35073 } }
I20260928 06:37:01.388501 17002 leader_election.cc:288] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe077003eb494bb78450f498da59abcd (127.5.22.3:43963), b4c83f7193c54f6ebc73d132e83c77f4 (127.5.22.2:35073)
I20260928 06:37:01.394271 16992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:01.389160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:49713 (local address 127.5.22.3:43963)
0928 06:37:01.389304 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:01.389308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:01.389827 (+ 519us) server_negotiation.cc:407] Connection header received
0928 06:37:01.389999 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:01.390006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:01.390073 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:01.390143 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:01.390654 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.391435 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.393062 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.393671 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.393790 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:01.393856 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:01.394074 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 06:37:01.394148 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 06:37:01.394330 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:01.388735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:43963 (local address 127.5.22.1:49713)
0928 06:37:01.389728 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:01.389740 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:01.389880 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:01.390274 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:01.390277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:01.390310 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:01.390506 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:01.390519 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.391592 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.391595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:01.392903 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:01.392913 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.393826 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.393833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:01.393937 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:01.393961 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 06:37:01.394037 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:37:01.394107 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"thread_start_us":129,"threads_started":1}
I20260928 06:37:01.394487 16989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:01.389642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:42185 (local address 127.5.22.2:35073)
0928 06:37:01.390131 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:01.390136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:01.390569 (+ 433us) server_negotiation.cc:407] Connection header received
0928 06:37:01.390756 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:01.390759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:01.390810 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:01.390876 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:01.391255 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.391967 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.393309 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.393785 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.393890 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:01.393956 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:01.394247 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 06:37:01.394372 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400}
I20260928 06:37:01.394845 17004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:01.389065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:35073 (local address 127.5.22.1:42185)
0928 06:37:01.390479 (+ 1414us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:01.390494 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:01.390641 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:01.391006 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:01.391008 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:01.391025 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:01.391134 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:01.391139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.392114 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.392118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:01.393196 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:01.393203 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.394013 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.394019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:01.394105 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:01.394125 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 06:37:01.394624 (+ 499us) client_negotiation.cc:240] Negotiation successful
0928 06:37:01.394695 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1347,"thread_start_us":107,"threads_started":1}
I20260928 06:37:01.395143 16930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25ef31f923c4c2c904aaa31a64c4d45" candidate_uuid: "71521cfa1c09413492f759488bcc529f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe077003eb494bb78450f498da59abcd" is_pre_election: true
I20260928 06:37:01.395490 16930 raft_consensus.cc:2454] T e25ef31f923c4c2c904aaa31a64c4d45 P fe077003eb494bb78450f498da59abcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71521cfa1c09413492f759488bcc529f in term 0.
I20260928 06:37:01.395509 16794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25ef31f923c4c2c904aaa31a64c4d45" candidate_uuid: "71521cfa1c09413492f759488bcc529f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4c83f7193c54f6ebc73d132e83c77f4" is_pre_election: true
I20260928 06:37:01.395798 16794 raft_consensus.cc:2454] T e25ef31f923c4c2c904aaa31a64c4d45 P b4c83f7193c54f6ebc73d132e83c77f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71521cfa1c09413492f759488bcc529f in term 0.
I20260928 06:37:01.396039 16593 leader_election.cc:302] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71521cfa1c09413492f759488bcc529f, fe077003eb494bb78450f498da59abcd; no voters:
I20260928 06:37:01.396322 17002 raft_consensus.cc:2781] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:37:01.396413 17002 raft_consensus.cc:482] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:37:01.396463 17002 raft_consensus.cc:3037] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:01.397498 17002 raft_consensus.cc:504] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71521cfa1c09413492f759488bcc529f" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 32931 } } peers { permanent_uuid: "fe077003eb494bb78450f498da59abcd" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43963 } } peers { permanent_uuid: "b4c83f7193c54f6ebc73d132e83c77f4" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35073 } }
I20260928 06:37:01.397821 17002 leader_election.cc:288] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f [CANDIDATE]: Term 1 election: Requested vote from peers fe077003eb494bb78450f498da59abcd (127.5.22.3:43963), b4c83f7193c54f6ebc73d132e83c77f4 (127.5.22.2:35073)
I20260928 06:37:01.398132 16794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25ef31f923c4c2c904aaa31a64c4d45" candidate_uuid: "71521cfa1c09413492f759488bcc529f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4c83f7193c54f6ebc73d132e83c77f4"
I20260928 06:37:01.398254 16794 raft_consensus.cc:3037] T e25ef31f923c4c2c904aaa31a64c4d45 P b4c83f7193c54f6ebc73d132e83c77f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:01.398566 16930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25ef31f923c4c2c904aaa31a64c4d45" candidate_uuid: "71521cfa1c09413492f759488bcc529f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe077003eb494bb78450f498da59abcd"
I20260928 06:37:01.398697 16930 raft_consensus.cc:3037] T e25ef31f923c4c2c904aaa31a64c4d45 P fe077003eb494bb78450f498da59abcd [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:01.399546 16794 raft_consensus.cc:2454] T e25ef31f923c4c2c904aaa31a64c4d45 P b4c83f7193c54f6ebc73d132e83c77f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71521cfa1c09413492f759488bcc529f in term 1.
I20260928 06:37:01.399727 16930 raft_consensus.cc:2454] T e25ef31f923c4c2c904aaa31a64c4d45 P fe077003eb494bb78450f498da59abcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71521cfa1c09413492f759488bcc529f in term 1.
I20260928 06:37:01.399853 16592 leader_election.cc:302] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71521cfa1c09413492f759488bcc529f, b4c83f7193c54f6ebc73d132e83c77f4; no voters:
I20260928 06:37:01.400012 17002 raft_consensus.cc:2781] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:01.400254 17002 raft_consensus.cc:686] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f [term 1 LEADER]: Becoming Leader. State: Replica: 71521cfa1c09413492f759488bcc529f, State: Running, Role: LEADER
I20260928 06:37:01.400624 17002 consensus_queue.cc:237] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71521cfa1c09413492f759488bcc529f" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 32931 } } peers { permanent_uuid: "fe077003eb494bb78450f498da59abcd" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43963 } } peers { permanent_uuid: "b4c83f7193c54f6ebc73d132e83c77f4" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35073 } }
I20260928 06:37:01.404071 16492 catalog_manager.cc:5495] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f reported cstate change: term changed from 0 to 1, leader changed from <none> to 71521cfa1c09413492f759488bcc529f (127.5.22.1). New cstate: current_term: 1 leader_uuid: "71521cfa1c09413492f759488bcc529f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71521cfa1c09413492f759488bcc529f" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 32931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe077003eb494bb78450f498da59abcd" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4c83f7193c54f6ebc73d132e83c77f4" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35073 } health_report { overall_health: UNKNOWN } } }
I20260928 06:37:01.429255 16993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:01.425087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45732 (local address 127.5.22.1:32931)
0928 06:37:01.425208 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:01.425213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:01.425326 (+ 113us) server_negotiation.cc:407] Connection header received
0928 06:37:01.425494 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:01.425496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:01.425556 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:01.425671 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:37:01.425930 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.426533 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.427677 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:01.427977 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:01.428091 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:01.428434 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:37:01.428867 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:37:01.429006 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:01.429051 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 06:37:01.429107 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:37:01.436480 16930 raft_consensus.cc:1261] T e25ef31f923c4c2c904aaa31a64c4d45 P fe077003eb494bb78450f498da59abcd [term 1 FOLLOWER]: Refusing update from remote peer 71521cfa1c09413492f759488bcc529f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:01.436480 16794 raft_consensus.cc:1261] T e25ef31f923c4c2c904aaa31a64c4d45 P b4c83f7193c54f6ebc73d132e83c77f4 [term 1 FOLLOWER]: Refusing update from remote peer 71521cfa1c09413492f759488bcc529f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:01.437444 17002 consensus_queue.cc:1035] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe077003eb494bb78450f498da59abcd" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 43963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:01.437732 17005 consensus_queue.cc:1035] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4c83f7193c54f6ebc73d132e83c77f4" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 35073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 06:37:01.446656 16705 tablet.cc:2369] T e25ef31f923c4c2c904aaa31a64c4d45 P 71521cfa1c09413492f759488bcc529f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:37:01.449215 17008 mvcc.cc:205] Tried to move back new op lower bound from 7334205118194700288 to 7334205118060191744. Current Snapshot: MvccSnapshot[applied={T|T < 7334205118194700288}]
I20260928 06:37:01.452116 17010 mvcc.cc:205] Tried to move back new op lower bound from 7334205118194700288 to 7334205118060191744. Current Snapshot: MvccSnapshot[applied={T|T < 7334205118194700288}]
I20260928 06:37:01.452438 17009 mvcc.cc:205] Tried to move back new op lower bound from 7334205118194700288 to 7334205118060191744. Current Snapshot: MvccSnapshot[applied={T|T < 7334205118194700288}]
I20260928 06:37:01.467990 16492 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:37:01.468192 16492 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:37:01.471263 16492 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2bc1b61539604a95aaaa9adcca273f8f: Sending DeleteTablet for 3 replicas of tablet e25ef31f923c4c2c904aaa31a64c4d45
I20260928 06:37:01.472366 16776 tablet_service.cc:1514] Processing DeleteTablet for tablet e25ef31f923c4c2c904aaa31a64c4d45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:01 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:44796
I20260928 06:37:01.472434 16638 tablet_service.cc:1514] Processing DeleteTablet for tablet e25ef31f923c4c2c904aaa31a64c4d45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:01 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:45726
I20260928 06:37:01.472905 16910 tablet_service.cc:1514] Processing DeleteTablet for tablet e25ef31f923c4c2c904aaa31a64c4d45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:01 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:39540
I20260928 06:37:01.473762 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 16573
I20260928 06:37:01.476166 17017 tablet_replica.cc:330] T e25ef31f923c4c2c904aaa31a64c4d45 P b4c83f7193c54f6ebc73d132e83c77f4: stopping tablet replica
I20260928 06:37:01.476567 17017 raft_consensus.cc:2229] T e25ef31f923c4c2c904aaa31a64c4d45 P b4c83f7193c54f6ebc73d132e83c77f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:37:01.477047 17017 raft_consensus.cc:2258] T e25ef31f923c4c2c904aaa31a64c4d45 P b4c83f7193c54f6ebc73d132e83c77f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:01.478938 17017 ts_tablet_manager.cc:1905] T e25ef31f923c4c2c904aaa31a64c4d45 P b4c83f7193c54f6ebc73d132e83c77f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:37:01.479000 17019 tablet_replica.cc:330] T e25ef31f923c4c2c904aaa31a64c4d45 P fe077003eb494bb78450f498da59abcd: stopping tablet replica
I20260928 06:37:01.479373 17019 raft_consensus.cc:2229] T e25ef31f923c4c2c904aaa31a64c4d45 P fe077003eb494bb78450f498da59abcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:37:01.479741 17019 raft_consensus.cc:2258] T e25ef31f923c4c2c904aaa31a64c4d45 P fe077003eb494bb78450f498da59abcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:01.482515 17017 ts_tablet_manager.cc:1918] T e25ef31f923c4c2c904aaa31a64c4d45 P b4c83f7193c54f6ebc73d132e83c77f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:37:01.482697 17017 log.cc:1196] T e25ef31f923c4c2c904aaa31a64c4d45 P b4c83f7193c54f6ebc73d132e83c77f4: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-1/wal/wals/e25ef31f923c4c2c904aaa31a64c4d45
I20260928 06:37:01.483162 17017 ts_tablet_manager.cc:1939] T e25ef31f923c4c2c904aaa31a64c4d45 P b4c83f7193c54f6ebc73d132e83c77f4: Deleting consensus metadata
I20260928 06:37:01.484383 17019 ts_tablet_manager.cc:1905] T e25ef31f923c4c2c904aaa31a64c4d45 P fe077003eb494bb78450f498da59abcd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:37:01.484655 16477 catalog_manager.cc:4841] TS b4c83f7193c54f6ebc73d132e83c77f4 (127.5.22.2:35073): tablet e25ef31f923c4c2c904aaa31a64c4d45 (table test-table [id=25be7d20a118442795af6b615baeb7b1]) successfully deleted
W20260928 06:37:01.487092 16478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.22.1:32931 (error 108)
I20260928 06:37:01.487133 17019 ts_tablet_manager.cc:1918] T e25ef31f923c4c2c904aaa31a64c4d45 P fe077003eb494bb78450f498da59abcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:37:01.487216 17019 log.cc:1196] T e25ef31f923c4c2c904aaa31a64c4d45 P fe077003eb494bb78450f498da59abcd: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/e25ef31f923c4c2c904aaa31a64c4d45
I20260928 06:37:01.487522 17019 ts_tablet_manager.cc:1939] T e25ef31f923c4c2c904aaa31a64c4d45 P fe077003eb494bb78450f498da59abcd: Deleting consensus metadata
I20260928 06:37:01.488859 16478 catalog_manager.cc:4841] TS fe077003eb494bb78450f498da59abcd (127.5.22.3:43963): tablet e25ef31f923c4c2c904aaa31a64c4d45 (table test-table [id=25be7d20a118442795af6b615baeb7b1]) successfully deleted
I20260928 06:37:01.488986 16988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:01.488701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:32931 (local address 127.0.0.1:45736)
0928 06:37:01.488829 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:01.488903 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.1:32931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 06:37:01.489185 16478 catalog_manager.cc:4584] TS 71521cfa1c09413492f759488bcc529f (127.5.22.1:32931): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e25ef31f923c4c2c904aaa31a64c4d45: Network error: Client connection negotiation failed: client connection to 127.5.22.1:32931: connect: Connection refused (error 111)
I20260928 06:37:01.489322 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 16712
I20260928 06:37:01.497300 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 16847
I20260928 06:37:01.504406 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 16460
I20260928 06:37:01.511109 5208 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 06:37:01.856106 5208 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 06:37:01.856251 5208 mini_ranger.cc:66] Stopping Ranger...
I20260928 06:37:02.201156 5208 mini_ranger.cc:68] Stopped Ranger
2026-09-28 06:37:02.201 UTC [16045] LOG: received smart shutdown request
2026-09-28 06:37:02.205 UTC [16045] LOG: background worker "logical replication launcher" (PID 16054) exited with exit code 1
2026-09-28 06:37:02.206 UTC [16049] LOG: shutting down
2026-09-28 06:37:02.348 UTC [16045] LOG: database system is shut down
2026-09-28T06:37:02Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49723 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:02 dist-test-slave-j0qf krb5kdc[17029](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:02 dist-test-slave-j0qf krb5kdc[17029](info): set up 2 sockets
Sep 28 06:37:02 dist-test-slave-j0qf krb5kdc[17029](info): commencing operation
krb5kdc: starting...
W20260928 06:37:03.533196 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:37:03 dist-test-slave-j0qf krb5kdc[17029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:03Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.22.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.22.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.22.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/rangeradmin_127.5.22.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.22.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/rangeradmin_127.5.22.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.22.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.22.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.22.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/rangerlookup_127.5.22.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.22.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/rangerlookup_127.5.22.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.22.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.22.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.22.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/HTTP_127.5.22.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.22.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/HTTP_127.5.22.20@KRBTEST.COM.keytab.
I20260928 06:37:03.705732 5208 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskgGsXow/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/postgres -l logfile start
2026-09-28 06:37:06.339 UTC [17059] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 06:37:06.339 UTC [17059] LOG: listening on IPv4 address "127.5.22.20", port 47317
2026-09-28 06:37:06.344 UTC [17059] LOG: listening on Unix socket "/tmp/.s.PGSQL.47317"
2026-09-28 06:37:06.367 UTC [17062] LOG: database system was shut down at 2026-09-28 06:37:04 UTC
2026-09-28 06:37:06.371 UTC [17059] LOG: database system is ready to accept connections
I20260928 06:37:07.344635 5208 mini_postgres.cc:96] Postgres bound to 47317
2026-09-28 06:37:07.351 UTC [17070] FATAL: database "slave" does not exist
127.5.22.20:47317 - accepting connections
I20260928 06:37:07.351835 5208 mini_ranger.cc:161] Starting Ranger...
I20260928 06:37:07.367117 5208 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 06:37:07.869436 5208 mini_ranger.cc:87] Created ranger Postgres database
I20260928 06:37:07.869524 5208 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ranger-admin
2026-09-28 06:37:08,141 [I] DB FLAVOR :POSTGRES
2026-09-28 06:37:08,141 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:37:08,142 [I] Checking connection..
2026-09-28 06:37:08,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:47317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 06:37:08,404 [I] Checking connection passed.
2026-09-28 06:37:08,405 [I] --------- Verifying version history table ---------
2026-09-28 06:37:08,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:47317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:37:08,655 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:37:08,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:47317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:37:08,897 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:37:08,897 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 06:37:08,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:47317/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 06:37:09.157 UTC [17134] WARNING: there is no transaction in progress
2026-09-28 06:37:09,163 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 06:37:09,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:47317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:37:09,416 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 06:37:09,417 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:37:09,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:47317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 06:37:09,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:47317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 06:37:09,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:47317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j0qf.c.gcp-upstream.internal','N') ;"
2026-09-28 06:37:10,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:47317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:37:10,376 [I] Table x_portal_user does not exist in database ranger
2026-09-28 06:37:10,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:47317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:37:10,644 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 06:37:10,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:47317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:37:10,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:47317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:37:11,122 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 06:37:11,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:47317/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 06:37:12.099 UTC [17242] WARNING: there is no transaction in progress
2026-09-28 06:37:12.116 UTC [17242] WARNING: there is no transaction in progress
2026-09-28 06:37:12.133 UTC [17242] WARNING: there is no transaction in progress
2026-09-28 06:37:12.149 UTC [17242] WARNING: there is no transaction in progress
2026-09-28 06:37:12.168 UTC [17242] WARNING: there is no transaction in progress
2026-09-28 06:37:12.185 UTC [17242] WARNING: there is no transaction in progress
2026-09-28 06:37:12.244 UTC [17242] WARNING: there is no transaction in progress
2026-09-28 06:37:12.251 UTC [17242] WARNING: there is no transaction in progress
2026-09-28 06:37:12.265 UTC [17242] WARNING: there is no transaction in progress
2026-09-28 06:37:12.276 UTC [17242] WARNING: there is no transaction in progress
2026-09-28 06:37:12.286 UTC [17242] WARNING: there is no transaction in progress
2026-09-28 06:37:12.302 UTC [17242] WARNING: there is no transaction in progress
2026-09-28 06:37:12.312 UTC [17242] WARNING: there is no transaction in progress
2026-09-28 06:37:12.319 UTC [17242] WARNING: there is no transaction in progress
2026-09-28 06:37:12.326 UTC [17242] WARNING: there is no transaction in progress
2026-09-28 06:37:12.334 UTC [17242] WARNING: there is no transaction in progress
2026-09-28 06:37:13,011 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 06:37:13,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:47317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 06:37:13,238 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 06:37:13,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:47317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:37:13,508 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 06:37:13,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:47317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:37:13,772 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 06:37:13,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:47317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:37:14,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:47317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:37:14,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:47317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j0qf.c.gcp-upstream.internal';"
2026-09-28 06:37:14,524 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 06:37:14,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:47317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 06:37:14,763 [I] DB_PATCHES have already been applied
I20260928 06:37:14.769073 5208 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ranger-admin:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskgGsXow/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 06:37:14.769137 5208 mini_ranger.cc:193] Using host: 127.5.22.20
I20260928 06:37:14.772817 5208 mini_ranger.cc:227] Ranger admin URL: http://127.5.22.20:53489
Sep 28, 2026 6:37:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:37:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:37:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 6:37:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 6:37:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 6:37:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 6:37:15 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 06:37:15.928031 5208 mini_ranger.cc:160] Time spent starting Ranger: real 8.576s user 0.000s sys 0.010s
Sep 28, 2026 6:37:35 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 6:37:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 6:37:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 6:37:35 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 6:37:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 6:37:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 6:37:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 6:37:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 6:37:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 6:37:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 6:37:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 6:37:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 6:37:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 6:37:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 6:37:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 6:37:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 6:37:35 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 6:37:36 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 06:37:36.716457 5208 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.22.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.22.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.22.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/rangerkms_127.5.22.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.22.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/rangerkms_127.5.22.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.22.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.22.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.22.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/HTTP_127.5.22.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.22.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/HTTP_127.5.22.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 06:37:36 dist-test-slave-j0qf krb5kdc[17029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577456, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 06:37:36.804903 5208 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 06:37:36.815279 5208 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 06:37:37.763824 5208 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 06:37:37.763955 5208 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ranger-kms
I20260928 06:37:37.763985 5208 mini_ranger_kms.cc:227] Using postgres at 127.5.22.20:47317
2026-09-28 06:37:37,942 [I] DB FLAVOR :POSTGRES
2026-09-28 06:37:37,942 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:37:37,942 [I] Checking connection
2026-09-28 06:37:37,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:47317/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 06:37:38,206 [I] connection success
2026-09-28 06:37:38,206 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 06:37:38,206 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 06:37:38,206 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:47317/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 06:37:38,524 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 06:37:38,524 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:37:38,525 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 06:37:38,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.22.20:47317/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 06:37:38,827 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 06:37:39.320218 5208 mini_ranger_kms.cc:310] Created kms service
I20260928 06:37:39.553274 5208 mini_ranger_kms.cc:326] Created kudu user
I20260928 06:37:39.598172 5208 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 06:37:39 dist-test-slave-j0qf krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577456, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.22.20@KRBTEST.COM
I20260928 06:37:40.027699 5208 mini_ranger_kms.cc:384] Added ranger policy
I20260928 06:37:40.027853 5208 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ranger-kms:/tmp/dist-test-taskgGsXow/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskgGsXow/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 06:37:40.027875 5208 mini_ranger_kms.cc:240] Using host: 127.5.22.20
I20260928 06:37:40.033142 5208 mini_ranger_kms.cc:276] Ranger KMS PID: 17422
I20260928 06:37:40.033265 5208 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.22.20:32803
Sep 28, 2026 6:37:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:37:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:37:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 6:37:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 6:37:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 6:37:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 06:37:42.337522 5208 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.533s user 0.007s sys 0.154s
I20260928 06:37:42.337651 5208 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 06:37:42.337715 5208 mini_ranger_kms.cc:401] 127.5.22.20:32803/kms/v1/keys
Sep 28 06:37:42 dist-test-slave-j0qf krb5kdc[17029](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577462, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.22.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 06:37:42 dist-test-slave-j0qf krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577462, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.22.20@KRBTEST.COM for HTTP/127.5.22.20@KRBTEST.COM
Sep 28 06:37:43 dist-test-slave-j0qf krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577462, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.22.20@KRBTEST.COM for HTTP/127.5.22.20@KRBTEST.COM
Sep 28 06:37:45 dist-test-slave-j0qf krb5kdc[17029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:37:45.673691 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:34401
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:34401
--ranger_config_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.22.20:32803/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:45.789882 17473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:45.790181 17473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:45.790222 17473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:45.794700 17473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:45.794778 17473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:45.794797 17473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:45.794814 17473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:45.794831 17473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:45.801015 17473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.22.20:32803/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:34401
--ranger_config_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:34401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:45.802105 17473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:45.803236 17473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:45.812135 17479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:45.812191 17478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:45.812186 17473 server_base.cc:1034] running on GCE node
W20260928 06:37:45.812415 17481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:45.813026 17473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:45.814175 17473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:45.815550 17473 hybrid_clock.cc:648] HybridClock initialized: now 1790577465815534 us; error 37 us; skew 500 ppm
Sep 28 06:37:45 dist-test-slave-j0qf krb5kdc[17029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577465, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:45.818641 17473 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:37:45.819650 17473 webserver.cc:460] Webserver started at http://127.5.22.62:39955/ using document root <none> and password file <none>
I20260928 06:37:45.820210 17473 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:45.820315 17473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:45.820557 17473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:37:45 dist-test-slave-j0qf krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577465, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for HTTP/127.5.22.20@KRBTEST.COM
I20260928 06:37:45.876269 17473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "5090a098891a48a1a23a7b88fd6f7a1f"
format_stamp: "Formatted at 2026-09-28 06:37:45 on dist-test-slave-j0qf"
server_key: "b7f3fc36c49ea06a18842989c88a3d5c"
server_key_iv: "b4b11e204cba85698a58c7a291cea845"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:45.877053 17473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "5090a098891a48a1a23a7b88fd6f7a1f"
format_stamp: "Formatted at 2026-09-28 06:37:45 on dist-test-slave-j0qf"
server_key: "b7f3fc36c49ea06a18842989c88a3d5c"
server_key_iv: "b4b11e204cba85698a58c7a291cea845"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:45.882170 17473 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 06:37:45.899807 17488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:45.901486 17473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:37:45.901641 17473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "5090a098891a48a1a23a7b88fd6f7a1f"
format_stamp: "Formatted at 2026-09-28 06:37:45 on dist-test-slave-j0qf"
server_key: "b7f3fc36c49ea06a18842989c88a3d5c"
server_key_iv: "b4b11e204cba85698a58c7a291cea845"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:45.901816 17473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:45.923897 17473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:45.927263 17473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:45.927477 17473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:45.936271 17473 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:34401
I20260928 06:37:45.936290 17540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:34401 every 8 connection(s)
I20260928 06:37:45.937489 17473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:45.940438 17541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:45.942679 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 17473
I20260928 06:37:45.942801 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:37:45.943099 5208 external_mini_cluster.cc:1428] Setting key 9dd9d61ceeb48a4032ae03a3e2a01776
I20260928 06:37:45.948278 17541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f: Bootstrap starting.
I20260928 06:37:45.951557 17541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:45.952653 17541 log.cc:824] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f: Log is configured to *not* fsync() on all Append() calls
Sep 28 06:37:45 dist-test-slave-j0qf krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:37:45.954881 17541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f: No bootstrap required, opened a new log
I20260928 06:37:45.959300 17541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5090a098891a48a1a23a7b88fd6f7a1f" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34401 } }
I20260928 06:37:45.959690 17541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:45.959786 17541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5090a098891a48a1a23a7b88fd6f7a1f, State: Initialized, Role: FOLLOWER
I20260928 06:37:45.960302 17541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5090a098891a48a1a23a7b88fd6f7a1f" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34401 } }
I20260928 06:37:45.960472 17541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:45.960556 17541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:45.960695 17541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:45.961180 17544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:45.944606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54886 (local address 127.5.22.62:34401)
0928 06:37:45.945162 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:45.945181 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:45.945225 (+ 44us) server_negotiation.cc:407] Connection header received
0928 06:37:45.946060 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:45.946082 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:45.946550 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:45.946921 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:45.948293 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:45.949809 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:45.951057 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:45.951736 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:45.951917 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:45.954970 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:45.954977 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:45.954994 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:45.955024 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:45.957706 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:45.958246 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:45.958252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:45.958258 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:45.958423 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:45.958740 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:45.958744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:45.958745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:45.959266 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:45.959467 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:45.959936 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 06:37:45.960214 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":113,"threads_started":1}
I20260928 06:37:45.961758 17541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5090a098891a48a1a23a7b88fd6f7a1f" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34401 } }
I20260928 06:37:45.962109 17541 leader_election.cc:302] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5090a098891a48a1a23a7b88fd6f7a1f; no voters:
I20260928 06:37:45.962524 17541 leader_election.cc:288] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:45.962641 17546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:45.962848 17546 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [term 1 LEADER]: Becoming Leader. State: Replica: 5090a098891a48a1a23a7b88fd6f7a1f, State: Running, Role: LEADER
I20260928 06:37:45.963197 17546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5090a098891a48a1a23a7b88fd6f7a1f" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34401 } }
I20260928 06:37:45.963836 17541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:45.964424 17546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5090a098891a48a1a23a7b88fd6f7a1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5090a098891a48a1a23a7b88fd6f7a1f" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34401 } } }
I20260928 06:37:45.964643 17546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:45.964898 17546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5090a098891a48a1a23a7b88fd6f7a1f. Latest consensus state: current_term: 1 leader_uuid: "5090a098891a48a1a23a7b88fd6f7a1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5090a098891a48a1a23a7b88fd6f7a1f" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34401 } } }
I20260928 06:37:45.965061 17546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:45.965358 17554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:45.968724 17554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:45.969151 17541 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 06:37:45.979475 17554 catalog_manager.cc:1347] Generated new cluster ID: df9cdee937ce4ce39ad0e74dc87cd5f9
I20260928 06:37:45.979638 17554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:46.028568 17554 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:46.030225 17554 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:46.037580 17554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f: Generated new TSK 0
I20260928 06:37:46.038512 17554 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.22.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
I20260928 06:37:46.710589 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.22.20:32803/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:34401
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:46.829986 17586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:46.830283 17586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:46.830402 17586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:46.834590 17586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:46.834666 17586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:46.834781 17586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:37:46.840605 17586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.22.20:32803/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:34401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:46.841742 17586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:46.842834 17586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:37:46.853672 17586 server_base.cc:1034] running on GCE node
W20260928 06:37:46.853710 17591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:46.853904 17594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:46.853962 17592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:46.854492 17586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:46.855268 17586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:46.856480 17586 hybrid_clock.cc:648] HybridClock initialized: now 1790577466856444 us; error 47 us; skew 500 ppm
Sep 28 06:37:46 dist-test-slave-j0qf krb5kdc[17029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577466, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:46.859937 17586 init.cc:377] Logged in from keytab as kudu/127.5.22.1@KRBTEST.COM (short username kudu)
I20260928 06:37:46.860895 17586 webserver.cc:460] Webserver started at http://127.5.22.1:34801/ using document root <none> and password file <none>
I20260928 06:37:46.861464 17586 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:46.861555 17586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:46.861761 17586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:37:46 dist-test-slave-j0qf krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577466, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for HTTP/127.5.22.20@KRBTEST.COM
I20260928 06:37:46.877871 17586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "ca05bc7ab5be4a2eaa6c0b8ddd561297"
format_stamp: "Formatted at 2026-09-28 06:37:46 on dist-test-slave-j0qf"
server_key: "f86226a7965ee6c74f15a1b3be9582f3"
server_key_iv: "d51837f52aa59edb512fd22351fb222d"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:46.878643 17586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "ca05bc7ab5be4a2eaa6c0b8ddd561297"
format_stamp: "Formatted at 2026-09-28 06:37:46 on dist-test-slave-j0qf"
server_key: "f86226a7965ee6c74f15a1b3be9582f3"
server_key_iv: "d51837f52aa59edb512fd22351fb222d"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:46.883797 17586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 06:37:46.906467 17603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:46.908325 17586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:37:46.908481 17586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "ca05bc7ab5be4a2eaa6c0b8ddd561297"
format_stamp: "Formatted at 2026-09-28 06:37:46 on dist-test-slave-j0qf"
server_key: "f86226a7965ee6c74f15a1b3be9582f3"
server_key_iv: "d51837f52aa59edb512fd22351fb222d"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:46.908622 17586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:46.935053 17586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:46.938735 17586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:46.939064 17586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:46.939791 17586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:46.940873 17586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:46.941064 17586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:46.941188 17586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:46.941236 17586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:46.951350 17586 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:40511
I20260928 06:37:46.952253 17586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:37:46.952561 17716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:40511 every 8 connection(s)
I20260928 06:37:46.959306 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 17586
I20260928 06:37:46.959445 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:37:46.959775 5208 external_mini_cluster.cc:1428] Setting key d2480c8dbc74cced653f8b9994bfa8d9
Sep 28 06:37:46 dist-test-slave-j0qf krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577466, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:37:46.982473 17719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:46.954735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:34401 (local address 127.5.22.1:54159)
0928 06:37:46.956530 (+ 1795us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:46.956587 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:46.957606 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:46.958565 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:46.958574 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:46.958925 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:46.959565 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:46.959583 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:46.960703 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:46.960706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:46.962074 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:46.962089 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:46.962951 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:46.962955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:46.963187 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:46.963837 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:46.963857 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:46.965656 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:46.969221 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:46.969229 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:46.969249 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:46.970841 (+ 1592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:46.979416 (+ 8575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:46.979420 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:46.979423 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:46.979695 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:46.980268 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:46.980276 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:46.980705 (+ 429us) client_negotiation.cc:769] Sending connection context
0928 06:37:46.980912 (+ 207us) client_negotiation.cc:240] Negotiation successful
0928 06:37:46.981295 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1567,"thread_start_us":143,"threads_started":1}
I20260928 06:37:46.984068 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:46.955807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:54159 (local address 127.5.22.62:34401)
0928 06:37:46.958028 (+ 2221us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:46.958037 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:46.958052 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:37:46.958095 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:46.958098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:46.958146 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:46.958231 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:46.959740 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:46.960563 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:46.962220 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:46.962660 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:46.962772 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:46.965839 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:46.965843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:46.965846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:46.965871 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:46.969033 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:46.979100 (+ 10067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:46.979106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:46.979108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:46.979249 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:46.979833 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:46.979836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:46.979838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:46.980106 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:46.980196 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:46.983746 (+ 3550us) server_negotiation.cc:299] Negotiation successful
0928 06:37:46.983899 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2143,"thread_start_us":106,"threads_started":1}
I20260928 06:37:46.985445 17717 heartbeater.cc:347] Connected to a master server at 127.5.22.62:34401
I20260928 06:37:46.985698 17717 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:46.986263 17717 heartbeater.cc:511] Master 127.5.22.62:34401 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.22.2@KRBTEST.COM; defaulting to no policy
I20260928 06:37:46.991523 17503 ts_manager.cc:194] Registered new tserver with Master: ca05bc7ab5be4a2eaa6c0b8ddd561297 (127.5.22.1:40511)
I20260928 06:37:46.993103 17503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.1@KRBTEST.COM'} at 127.5.22.1:54159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
I20260928 06:37:47.015537 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.22.20:32803/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:34401
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:37:47 dist-test-slave-j0qf krb5kdc[17029](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 06:37:47.153802 17725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:47.154181 17725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:47.154250 17725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:47.158877 17725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:47.158941 17725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:47.159047 17725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:37:47.165449 17725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.22.20:32803/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:34401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:47.166847 17725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:47.167927 17725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:47.177836 17731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:47.177836 17732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:47.178195 17725 server_base.cc:1034] running on GCE node
W20260928 06:37:47.178370 17734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:47.178812 17725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:47.179534 17725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:47.180745 17725 hybrid_clock.cc:648] HybridClock initialized: now 1790577467180715 us; error 41 us; skew 500 ppm
Sep 28 06:37:47 dist-test-slave-j0qf krb5kdc[17029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:47.183718 17725 init.cc:377] Logged in from keytab as kudu/127.5.22.2@KRBTEST.COM (short username kudu)
I20260928 06:37:47.184834 17725 webserver.cc:460] Webserver started at http://127.5.22.2:34391/ using document root <none> and password file <none>
I20260928 06:37:47.185511 17725 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:47.185607 17725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:47.185837 17725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:37:47 dist-test-slave-j0qf krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for HTTP/127.5.22.20@KRBTEST.COM
I20260928 06:37:47.201493 17725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "6064a462f63a468b896673c4cf6629e7"
format_stamp: "Formatted at 2026-09-28 06:37:47 on dist-test-slave-j0qf"
server_key: "c3b1b8a07add98189f513518c877b7f3"
server_key_iv: "04856ea4265d43d2f6779e12a1d646dc"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:47.202126 17725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "6064a462f63a468b896673c4cf6629e7"
format_stamp: "Formatted at 2026-09-28 06:37:47 on dist-test-slave-j0qf"
server_key: "c3b1b8a07add98189f513518c877b7f3"
server_key_iv: "04856ea4265d43d2f6779e12a1d646dc"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:47.206094 17725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260928 06:37:47.225302 17741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:47.227149 17725 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 06:37:47.227305 17725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "6064a462f63a468b896673c4cf6629e7"
format_stamp: "Formatted at 2026-09-28 06:37:47 on dist-test-slave-j0qf"
server_key: "c3b1b8a07add98189f513518c877b7f3"
server_key_iv: "04856ea4265d43d2f6779e12a1d646dc"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:47.227468 17725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:47.245569 17725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:47.249696 17725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:47.249922 17725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:47.250717 17725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:47.251821 17725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:47.251901 17725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:47.251960 17725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:47.251991 17725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:47.266449 17725 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:42785
I20260928 06:37:47.267416 17725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:37:47.268538 17854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:42785 every 8 connection(s)
I20260928 06:37:47.269919 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 17725
I20260928 06:37:47.270038 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:37:47.270398 5208 external_mini_cluster.cc:1428] Setting key e99b928a50f7b232b57b1f32e25d9dd9
Sep 28 06:37:47 dist-test-slave-j0qf krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.22.3@KRBTEST.COM; defaulting to no policy
I20260928 06:37:47.299458 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.269509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:36637 (local address 127.5.22.62:34401)
0928 06:37:47.269640 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.269645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.278902 (+ 9257us) server_negotiation.cc:407] Connection header received
0928 06:37:47.280173 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.280177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.280235 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.280332 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.281918 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.282817 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.285392 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.285775 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.285874 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.294756 (+ 8882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:47.294761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:47.294764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:47.294799 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:47.296698 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.297411 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.297416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.297418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.297504 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.297950 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.297955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.297957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.298248 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:47.298350 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:47.299105 (+ 755us) server_negotiation.cc:299] Negotiation successful
0928 06:37:47.299239 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:37:47.301183 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.270009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:34401 (local address 127.5.22.2:36637)
0928 06:37:47.278707 (+ 8698us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.278778 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:47.279933 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:47.280672 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:47.280682 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:47.281049 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:47.281734 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:47.281753 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.282985 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.282988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:47.284464 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:47.284475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.286672 (+ 2197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.286676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:47.286886 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.287629 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:47.287652 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:47.294512 (+ 6860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:47.296858 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:47.296867 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:47.296884 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:47.297264 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:47.297623 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:47.297626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:47.297628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:47.297829 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:47.298456 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:47.298462 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:47.298820 (+ 358us) client_negotiation.cc:769] Sending connection context
0928 06:37:47.299728 (+ 908us) client_negotiation.cc:240] Negotiation successful
0928 06:37:47.300091 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8433,"thread_start_us":145,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.3@KRBTEST.COM" created.
I20260928 06:37:47.302544 17855 heartbeater.cc:347] Connected to a master server at 127.5.22.62:34401
I20260928 06:37:47.302793 17855 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:47.303403 17855 heartbeater.cc:511] Master 127.5.22.62:34401 requested a full tablet report, sending...
I20260928 06:37:47.304522 17503 ts_manager.cc:194] Registered new tserver with Master: 6064a462f63a468b896673c4cf6629e7 (127.5.22.2:42785)
I20260928 06:37:47.305295 17503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.2@KRBTEST.COM'} at 127.5.22.2:36637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
I20260928 06:37:47.328187 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.22.20:32803/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:34401
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:47.508234 17860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:47.508615 17860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:47.508706 17860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:47.514941 17860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:47.515028 17860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:47.515112 17860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:37:47.522681 17860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.22.20:32803/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:34401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:47.524013 17860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:47.525329 17860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:47.537601 17866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:47.538112 17868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:47.538466 17865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:47.539304 17860 server_base.cc:1034] running on GCE node
I20260928 06:37:47.539856 17860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:47.540604 17860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:47.541803 17860 hybrid_clock.cc:648] HybridClock initialized: now 1790577467541734 us; error 82 us; skew 500 ppm
Sep 28 06:37:47 dist-test-slave-j0qf krb5kdc[17029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:47.545423 17860 init.cc:377] Logged in from keytab as kudu/127.5.22.3@KRBTEST.COM (short username kudu)
I20260928 06:37:47.546546 17860 webserver.cc:460] Webserver started at http://127.5.22.3:39545/ using document root <none> and password file <none>
I20260928 06:37:47.547353 17860 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:47.547482 17860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:47.547744 17860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:37:47 dist-test-slave-j0qf krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for HTTP/127.5.22.20@KRBTEST.COM
I20260928 06:37:47.572536 17860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "cf1f77ab5c25419eb63d701ce814eb5c"
format_stamp: "Formatted at 2026-09-28 06:37:47 on dist-test-slave-j0qf"
server_key: "495f5caca1f150e89358f2cbef1d3431"
server_key_iv: "75160c830a687480ca36db6d6171038c"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:47.579813 17860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "cf1f77ab5c25419eb63d701ce814eb5c"
format_stamp: "Formatted at 2026-09-28 06:37:47 on dist-test-slave-j0qf"
server_key: "495f5caca1f150e89358f2cbef1d3431"
server_key_iv: "75160c830a687480ca36db6d6171038c"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:47.593354 17860 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.000s sys 0.004s
I20260928 06:37:47.634770 17875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:47.636610 17860 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.004s sys 0.000s
I20260928 06:37:47.636749 17860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "cf1f77ab5c25419eb63d701ce814eb5c"
format_stamp: "Formatted at 2026-09-28 06:37:47 on dist-test-slave-j0qf"
server_key: "495f5caca1f150e89358f2cbef1d3431"
server_key_iv: "75160c830a687480ca36db6d6171038c"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:47.636909 17860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 28 06:37:47 dist-test-slave-j0qf krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for HTTP/127.5.22.20@KRBTEST.COM
I20260928 06:37:47.688346 17860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:47.692287 17860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:47.692512 17860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:47.693377 17860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:47.694551 17860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:47.694624 17860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:47.694892 17860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:47.695029 17860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:47.718811 17860 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:40815
I20260928 06:37:47.720786 17860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:37:47.723737 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 17860
I20260928 06:37:47.723878 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
I20260928 06:37:47.724220 5208 external_mini_cluster.cc:1428] Setting key 637576868bdb7ac2b972d8e1c5371e1b
I20260928 06:37:47.735330 17988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:40815 every 8 connection(s)
Sep 28 06:37:47 dist-test-slave-j0qf krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:37:47.776371 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.746375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:36415 (local address 127.5.22.62:34401)
0928 06:37:47.746684 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.746689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.759508 (+ 12819us) server_negotiation.cc:407] Connection header received
0928 06:37:47.761400 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.761404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.761481 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.761538 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.763692 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.764473 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.766602 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.766967 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.767041 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.770342 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:47.770347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:47.770350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:47.770383 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:47.773350 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.774147 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.774152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.774154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.774252 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.774750 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.774754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.774755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.775010 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:47.775100 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:47.776025 (+ 925us) server_negotiation.cc:299] Negotiation successful
0928 06:37:47.776181 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213}
I20260928 06:37:47.782704 17991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.744828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:34401 (local address 127.5.22.3:36415)
0928 06:37:47.759153 (+ 14325us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.759259 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:47.761067 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:47.762162 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:47.762176 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:47.762697 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:47.763477 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:47.763498 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.764703 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.764707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:47.766455 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:47.766468 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.767278 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.767282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:47.767502 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.768258 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:47.768282 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:47.770078 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:47.773557 (+ 3479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:47.773565 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:47.773587 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:47.774012 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:47.774396 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:47.774399 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:47.774401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:47.774625 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:47.775173 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:47.775180 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:47.775562 (+ 382us) client_negotiation.cc:769] Sending connection context
0928 06:37:47.776868 (+ 1306us) client_negotiation.cc:240] Negotiation successful
0928 06:37:47.777303 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13858,"thread_start_us":167,"threads_started":1}
I20260928 06:37:47.784730 17989 heartbeater.cc:347] Connected to a master server at 127.5.22.62:34401
I20260928 06:37:47.785048 17989 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:47.785704 17989 heartbeater.cc:511] Master 127.5.22.62:34401 requested a full tablet report, sending...
I20260928 06:37:47.787118 17503 ts_manager.cc:194] Registered new tserver with Master: cf1f77ab5c25419eb63d701ce814eb5c (127.5.22.3:40815)
I20260928 06:37:47.788010 17503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.3@KRBTEST.COM'} at 127.5.22.3:36415
I20260928 06:37:47.789526 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:37:47.790645 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 17586
I20260928 06:37:47.797866 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 17725
I20260928 06:37:47.807523 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 17860
I20260928 06:37:47.814558 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 17473
I20260928 06:37:47.823802 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:34401
--webserver_interface=127.5.22.62
--webserver_port=39955
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:34401
--ranger_config_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.22.20:32803/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.22.20:32803/kms with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:47.959283 17992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:47.959579 17992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:47.959621 17992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:47.964465 17992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:47.964542 17992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:47.964561 17992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:47.964578 17992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:47.964594 17992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:47.971578 17992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.22.20:32803/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:34401
--ranger_config_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:34401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=39955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:47.972903 17992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:47.974013 17992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:47.983201 18000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:47.983201 17998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:47.983243 17992 server_base.cc:1034] running on GCE node
W20260928 06:37:47.983197 17997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:47.984197 17992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:47.985358 17992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:47.986590 17992 hybrid_clock.cc:648] HybridClock initialized: now 1790577467986575 us; error 29 us; skew 500 ppm
Sep 28 06:37:47 dist-test-slave-j0qf krb5kdc[17029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:47.989818 17992 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:37:47.991053 17992 webserver.cc:460] Webserver started at http://127.5.22.62:39955/ using document root <none> and password file <none>
I20260928 06:37:47.991634 17992 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:47.991709 17992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:37:48 dist-test-slave-j0qf krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for HTTP/127.5.22.20@KRBTEST.COM
I20260928 06:37:48.056681 17992 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 06:37:48.059706 18008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:48.061573 17992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:37:48.061698 17992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "5090a098891a48a1a23a7b88fd6f7a1f"
format_stamp: "Formatted at 2026-09-28 06:37:45 on dist-test-slave-j0qf"
server_key: "b7f3fc36c49ea06a18842989c88a3d5c"
server_key_iv: "b4b11e204cba85698a58c7a291cea845"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:48.062451 17992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:48.088397 17992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:48.091688 17992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:48.091949 17992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:48.099738 17992 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:34401
I20260928 06:37:48.099818 18060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:34401 every 8 connection(s)
I20260928 06:37:48.100859 17992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:48.104310 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 17992
I20260928 06:37:48.104772 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.22.20:32803/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:40511
--local_ip_for_outbound_sockets=127.5.22.1
--tserver_master_addrs=127.5.22.62:34401
--webserver_port=34801
--webserver_interface=127.5.22.1
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.22.20:32803/kms with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:37:48.105237 18061 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:37:48.106866 18061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f: Bootstrap starting.
I20260928 06:37:48.123922 18061 log.cc:824] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:48.130406 18061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:37:48.130807 18061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f: Bootstrap complete.
I20260928 06:37:48.134953 18061 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5090a098891a48a1a23a7b88fd6f7a1f" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34401 } }
I20260928 06:37:48.135370 18061 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5090a098891a48a1a23a7b88fd6f7a1f, State: Initialized, Role: FOLLOWER
I20260928 06:37:48.135914 18061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5090a098891a48a1a23a7b88fd6f7a1f" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34401 } }
I20260928 06:37:48.136058 18061 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:48.136147 18061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:48.136263 18061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:37:48.138163 18061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5090a098891a48a1a23a7b88fd6f7a1f" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34401 } }
I20260928 06:37:48.138527 18061 leader_election.cc:302] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5090a098891a48a1a23a7b88fd6f7a1f; no voters:
I20260928 06:37:48.138870 18061 leader_election.cc:288] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:37:48.138960 18065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:37:48.139202 18065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [term 2 LEADER]: Becoming Leader. State: Replica: 5090a098891a48a1a23a7b88fd6f7a1f, State: Running, Role: LEADER
I20260928 06:37:48.139564 18065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5090a098891a48a1a23a7b88fd6f7a1f" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34401 } }
I20260928 06:37:48.140120 18061 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:48.140560 18067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5090a098891a48a1a23a7b88fd6f7a1f. Latest consensus state: current_term: 2 leader_uuid: "5090a098891a48a1a23a7b88fd6f7a1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5090a098891a48a1a23a7b88fd6f7a1f" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34401 } } }
I20260928 06:37:48.140687 18067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:48.140940 18066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5090a098891a48a1a23a7b88fd6f7a1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5090a098891a48a1a23a7b88fd6f7a1f" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34401 } } }
I20260928 06:37:48.141069 18066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:48.141414 18069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:48.144644 18069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:48.145171 18061 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 06:37:48.145370 18069 catalog_manager.cc:1259] Loaded cluster ID: df9cdee937ce4ce39ad0e74dc87cd5f9
I20260928 06:37:48.145488 18069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:48.147828 18069 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:48.148545 18069 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f: Loaded TSK: 0
I20260928 06:37:48.149714 18069 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:37:48.250728 18063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:48.251089 18063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:48.251160 18063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:48.257357 18063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:48.257514 18063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:48.257664 18063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:37:48.265790 18063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.22.20:32803/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:40511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=34801
--tserver_master_addrs=127.5.22.62:34401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:48.267131 18063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:48.268455 18063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:48.281373 18098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:48.281478 18099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:48.282243 18063 server_base.cc:1034] running on GCE node
W20260928 06:37:48.282092 18101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:48.282956 18063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:48.283769 18063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:48.284968 18063 hybrid_clock.cc:648] HybridClock initialized: now 1790577468284934 us; error 49 us; skew 500 ppm
Sep 28 06:37:48 dist-test-slave-j0qf krb5kdc[17029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577468, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:48.288632 18063 init.cc:377] Logged in from keytab as kudu/127.5.22.1@KRBTEST.COM (short username kudu)
I20260928 06:37:48.289659 18063 webserver.cc:460] Webserver started at http://127.5.22.1:34801/ using document root <none> and password file <none>
I20260928 06:37:48.290477 18063 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:48.290637 18063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:37:48 dist-test-slave-j0qf krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577468, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for HTTP/127.5.22.20@KRBTEST.COM
I20260928 06:37:48.319938 18063 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.003s sys 0.000s
I20260928 06:37:48.323323 18109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:48.324939 18063 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 06:37:48.325059 18063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "ca05bc7ab5be4a2eaa6c0b8ddd561297"
format_stamp: "Formatted at 2026-09-28 06:37:46 on dist-test-slave-j0qf"
server_key: "f86226a7965ee6c74f15a1b3be9582f3"
server_key_iv: "d51837f52aa59edb512fd22351fb222d"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:48.325732 18063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:48.346140 18063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:48.349619 18063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:48.349838 18063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:48.350533 18063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:48.351477 18063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:48.351544 18063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:48.351609 18063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:48.351635 18063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:48.365451 18063 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:40511
I20260928 06:37:48.365571 18224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:40511 every 8 connection(s)
I20260928 06:37:48.366714 18063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:37:48.373101 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 18063
I20260928 06:37:48.373592 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.22.20:32803/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:42785
--local_ip_for_outbound_sockets=127.5.22.2
--tserver_master_addrs=127.5.22.62:34401
--webserver_port=34391
--webserver_interface=127.5.22.2
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.22.20:32803/kms with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:37:48 dist-test-slave-j0qf krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577468, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:37:48.401930 18228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:48.370995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:39105 (local address 127.5.22.62:34401)
0928 06:37:48.374560 (+ 3565us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:48.374584 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:48.374631 (+ 47us) server_negotiation.cc:407] Connection header received
0928 06:37:48.374928 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:48.374944 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:48.375404 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:48.375758 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:48.377037 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:48.382827 (+ 5790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:48.384732 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:48.385516 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:48.387562 (+ 2046us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:48.394125 (+ 6563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:48.394133 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:48.394154 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:48.394194 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:48.396504 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:48.397238 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:48.397245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:48.397251 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:48.397399 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:48.397848 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:48.397852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:48.397854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:48.398408 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:48.398549 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:48.400521 (+ 1972us) server_negotiation.cc:299] Negotiation successful
0928 06:37:48.400878 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3381,"thread_start_us":129,"threads_started":1}
I20260928 06:37:48.402670 18227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:48.369469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:34401 (local address 127.5.22.1:39105)
0928 06:37:48.372722 (+ 3253us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:48.372783 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:48.373996 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:48.375888 (+ 1892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:48.375897 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:48.376240 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:48.376832 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:48.376847 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:48.383023 (+ 6176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:48.383027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:48.384553 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:48.384568 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:48.385718 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:48.385723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:48.385927 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:48.386723 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:48.386746 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:48.393862 (+ 7116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:48.396670 (+ 2808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:48.396678 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:48.396696 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:48.397084 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:48.397502 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:48.397505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:48.397507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:48.397725 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:48.398565 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:48.398572 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:48.398906 (+ 334us) client_negotiation.cc:769] Sending connection context
0928 06:37:48.399112 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 06:37:48.399471 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2997,"thread_start_us":151,"threads_started":1}
I20260928 06:37:48.405510 18225 heartbeater.cc:347] Connected to a master server at 127.5.22.62:34401
I20260928 06:37:48.405756 18225 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:48.406385 18225 heartbeater.cc:511] Master 127.5.22.62:34401 requested a full tablet report, sending...
W20260928 06:37:48.408638 18225 heartbeater.cc:644] Failed to heartbeat to 127.5.22.62:34401 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 06:37:48.408763 18225 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:48.408942 18225 heartbeater.cc:511] Master 127.5.22.62:34401 requested a full tablet report, sending...
I20260928 06:37:48.409842 18225 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:48.410024 18225 heartbeater.cc:511] Master 127.5.22.62:34401 requested a full tablet report, sending...
W20260928 06:37:48.410861 18225 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:37:48.522447 18229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:48.522763 18229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:48.522827 18229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:48.528458 18229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:48.528524 18229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:48.528611 18229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:37:48.534335 18229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.22.20:32803/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:42785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=34391
--tserver_master_addrs=127.5.22.62:34401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:48.535473 18229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:48.536813 18229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:48.547986 18237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:48.548166 18229 server_base.cc:1034] running on GCE node
W20260928 06:37:48.548063 18234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:48.548024 18235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:48.548849 18229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:48.549520 18229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:48.550689 18229 hybrid_clock.cc:648] HybridClock initialized: now 1790577468550675 us; error 40 us; skew 500 ppm
Sep 28 06:37:48 dist-test-slave-j0qf krb5kdc[17029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577468, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:48.553517 18229 init.cc:377] Logged in from keytab as kudu/127.5.22.2@KRBTEST.COM (short username kudu)
I20260928 06:37:48.554577 18229 webserver.cc:460] Webserver started at http://127.5.22.2:34391/ using document root <none> and password file <none>
I20260928 06:37:48.555143 18229 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:48.555219 18229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:37:48 dist-test-slave-j0qf krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577468, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for HTTP/127.5.22.20@KRBTEST.COM
I20260928 06:37:48.582801 18229 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 06:37:48.586380 18245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:48.588075 18229 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 06:37:48.588210 18229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "6064a462f63a468b896673c4cf6629e7"
format_stamp: "Formatted at 2026-09-28 06:37:47 on dist-test-slave-j0qf"
server_key: "c3b1b8a07add98189f513518c877b7f3"
server_key_iv: "04856ea4265d43d2f6779e12a1d646dc"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:48.588950 18229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:48.625648 18229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:48.629426 18229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:48.629626 18229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:48.630470 18229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:48.631582 18229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:48.631644 18229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:48.631695 18229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:48.631726 18229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:48.642725 18229 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:42785
I20260928 06:37:48.643980 18229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:37:48.645357 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 18229
I20260928 06:37:48.645906 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.22.20:32803/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:40815
--local_ip_for_outbound_sockets=127.5.22.3
--tserver_master_addrs=127.5.22.62:34401
--webserver_port=39545
--webserver_interface=127.5.22.3
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.22.20:32803/kms with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:37:48.646309 18360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:42785 every 8 connection(s)
Sep 28 06:37:48 dist-test-slave-j0qf krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577468, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:37:48.662552 18228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:48.650234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:51291 (local address 127.5.22.62:34401)
0928 06:37:48.650391 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:48.650396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:48.650892 (+ 496us) server_negotiation.cc:407] Connection header received
0928 06:37:48.651947 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:48.651951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:48.652013 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:48.652070 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:48.653506 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:48.654371 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:48.655588 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:48.655835 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:48.655965 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:48.658648 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:48.658652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:48.658656 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:48.658705 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:48.660288 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:48.660867 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:48.660871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:48.660873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:48.660959 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:48.661320 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:48.661324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:48.661326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:48.661520 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:48.661632 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:48.662194 (+ 562us) server_negotiation.cc:299] Negotiation successful
0928 06:37:48.662370 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 06:37:48.663545 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:48.649088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:34401 (local address 127.5.22.2:51291)
0928 06:37:48.650699 (+ 1611us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:48.650754 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:48.651682 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:48.652495 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:48.652502 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:48.652773 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:48.653314 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:48.653335 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:48.654533 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:48.654535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:48.655443 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:48.655452 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:48.656006 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:48.656010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:48.656188 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:48.656834 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:48.656850 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:48.658431 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:48.660442 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:48.660449 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:48.660465 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:48.660749 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:48.661054 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:48.661057 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:48.661060 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:48.661220 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:48.661631 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:48.661640 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:48.661937 (+ 297us) client_negotiation.cc:769] Sending connection context
0928 06:37:48.662138 (+ 201us) client_negotiation.cc:240] Negotiation successful
0928 06:37:48.662511 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1393,"thread_start_us":184,"threads_started":1}
I20260928 06:37:48.665104 18361 heartbeater.cc:347] Connected to a master server at 127.5.22.62:34401
I20260928 06:37:48.665345 18361 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:48.665875 18361 heartbeater.cc:511] Master 127.5.22.62:34401 requested a full tablet report, sending...
W20260928 06:37:48.667672 18361 heartbeater.cc:644] Failed to heartbeat to 127.5.22.62:34401 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 06:37:48.667797 18361 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:48.667975 18361 heartbeater.cc:511] Master 127.5.22.62:34401 requested a full tablet report, sending...
I20260928 06:37:48.668815 18361 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:48.669018 18361 heartbeater.cc:511] Master 127.5.22.62:34401 requested a full tablet report, sending...
W20260928 06:37:48.669793 18361 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:37:48.808694 18363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:48.809012 18363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:48.809126 18363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:48.813362 18363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:48.813457 18363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:48.813561 18363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:37:48.821841 18363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.22.20:32803/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:40815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=39545
--tserver_master_addrs=127.5.22.62:34401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:48.823472 18363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:48.824548 18363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:48.834954 18372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:48.835461 18370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:48.835995 18363 server_base.cc:1034] running on GCE node
W20260928 06:37:48.836186 18369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:48.836689 18363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:48.837420 18363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:48.840690 18363 hybrid_clock.cc:648] HybridClock initialized: now 1790577468840649 us; error 59 us; skew 500 ppm
Sep 28 06:37:48 dist-test-slave-j0qf krb5kdc[17029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577468, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:48.844614 18363 init.cc:377] Logged in from keytab as kudu/127.5.22.3@KRBTEST.COM (short username kudu)
I20260928 06:37:48.845682 18363 webserver.cc:460] Webserver started at http://127.5.22.3:39545/ using document root <none> and password file <none>
I20260928 06:37:48.846268 18363 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:48.846405 18363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:37:48 dist-test-slave-j0qf krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577468, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for HTTP/127.5.22.20@KRBTEST.COM
I20260928 06:37:48.867825 18363 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.002s
I20260928 06:37:48.872064 18380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:48.873651 18363 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 06:37:48.873773 18363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "cf1f77ab5c25419eb63d701ce814eb5c"
format_stamp: "Formatted at 2026-09-28 06:37:47 on dist-test-slave-j0qf"
server_key: "495f5caca1f150e89358f2cbef1d3431"
server_key_iv: "75160c830a687480ca36db6d6171038c"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:48.874655 18363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:48.901319 18363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:48.905148 18363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:48.905416 18363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:48.906186 18363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:48.907218 18363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:48.907336 18363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:48.907441 18363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:48.907511 18363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:48.918695 18363 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:40815
I20260928 06:37:48.918741 18494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:40815 every 8 connection(s)
I20260928 06:37:48.919739 18363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:37:48.921900 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 18363
Sep 28 06:37:48 dist-test-slave-j0qf krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577468, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:37:48.936410 18228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:48.921829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:46123 (local address 127.5.22.62:34401)
0928 06:37:48.921970 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:48.921974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:48.923378 (+ 1404us) server_negotiation.cc:407] Connection header received
0928 06:37:48.924678 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:48.924681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:48.924738 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:48.924862 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:48.926467 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:48.927368 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:48.928831 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:48.929178 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:48.929271 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:48.932709 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:48.932716 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:48.932718 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:48.932768 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:48.934222 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:48.934791 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:48.934794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:48.934796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:48.934886 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:48.935260 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:48.935263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:48.935265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:48.935455 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:48.935563 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:48.936116 (+ 553us) server_negotiation.cc:299] Negotiation successful
0928 06:37:48.936238 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":43}
I20260928 06:37:48.937355 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:48.922463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:34401 (local address 127.5.22.3:46123)
0928 06:37:48.923167 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:48.923239 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:48.924313 (+ 1074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:48.925205 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:48.925215 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:48.925664 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:48.926164 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:48.926177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:48.927507 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:48.927511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:48.928655 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:48.928664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:48.929467 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:48.929471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:48.929667 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:48.930454 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:48.930495 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:48.932497 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:48.934385 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:48.934391 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:48.934407 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:48.934678 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:48.934982 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:48.934986 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:48.934988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:48.935154 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:48.935563 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:48.935571 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:48.935865 (+ 294us) client_negotiation.cc:769] Sending connection context
0928 06:37:48.936063 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 06:37:48.936417 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":152,"threads_started":1}
I20260928 06:37:48.938732 18495 heartbeater.cc:347] Connected to a master server at 127.5.22.62:34401
I20260928 06:37:48.939002 18495 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:48.939606 18495 heartbeater.cc:511] Master 127.5.22.62:34401 requested a full tablet report, sending...
I20260928 06:37:48.941092 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:48.922651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39916 (local address 127.5.22.62:34401)
0928 06:37:48.924762 (+ 2111us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:48.924775 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:48.925210 (+ 435us) server_negotiation.cc:407] Connection header received
0928 06:37:48.925255 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:48.925259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:48.925322 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:48.925411 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:48.926457 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:48.927113 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:48.928318 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:48.928655 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:48.928763 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:48.930930 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:48.930936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:48.930940 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:48.930972 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:48.933244 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:48.938860 (+ 5616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:48.938864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:48.938865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:48.938954 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:48.939331 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:48.939335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:48.939337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:48.939570 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:48.939783 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:48.940736 (+ 953us) server_negotiation.cc:299] Negotiation successful
0928 06:37:48.940890 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":2032,"thread_start_us":89,"threads_started":1}
W20260928 06:37:48.941624 18495 heartbeater.cc:644] Failed to heartbeat to 127.5.22.62:34401 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 06:37:48.941751 18495 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:48.941923 18495 heartbeater.cc:511] Master 127.5.22.62:34401 requested a full tablet report, sending...
I20260928 06:37:48.943619 18025 ts_manager.cc:194] Registered new tserver with Master: cf1f77ab5c25419eb63d701ce814eb5c (127.5.22.3:40815)
I20260928 06:37:48.945211 18025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.3@KRBTEST.COM'} at 127.5.22.3:46123
Sep 28 06:37:49 dist-test-slave-j0qf krb5kdc[17029](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577469, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:49.411093 18225 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:49.411405 18225 heartbeater.cc:511] Master 127.5.22.62:34401 requested a full tablet report, sending...
I20260928 06:37:49.412125 18025 ts_manager.cc:194] Registered new tserver with Master: ca05bc7ab5be4a2eaa6c0b8ddd561297 (127.5.22.1:40511)
I20260928 06:37:49.412761 18025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.1@KRBTEST.COM'} at 127.5.22.1:39105
I20260928 06:37:49.670046 18361 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:49.670374 18361 heartbeater.cc:511] Master 127.5.22.62:34401 requested a full tablet report, sending...
I20260928 06:37:49.671119 18025 ts_manager.cc:194] Registered new tserver with Master: 6064a462f63a468b896673c4cf6629e7 (127.5.22.2:42785)
I20260928 06:37:49.671891 18025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.2@KRBTEST.COM'} at 127.5.22.2:51291
I20260928 06:37:49.672310 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:37:49.684916 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:49.674267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39930 (local address 127.5.22.62:34401)
0928 06:37:49.674513 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:49.674518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:49.675449 (+ 931us) server_negotiation.cc:407] Connection header received
0928 06:37:49.675564 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:49.675567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:49.675620 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:49.675712 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:49.676576 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:49.677376 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:49.678614 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:49.678980 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:49.679055 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:49.680935 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:49.680939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:49.680942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:49.680969 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:49.682843 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:49.683364 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:49.683368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:49.683369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:49.683484 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:49.683847 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:49.683850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:49.683852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:49.684074 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:49.684185 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:49.684599 (+ 414us) server_negotiation.cc:299] Negotiation successful
0928 06:37:49.684727 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":91,"threads_started":1}
I20260928 06:37:49.688961 18025 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:37:49.695016 18025 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 28 06:37:49 dist-test-slave-j0qf krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577469, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for HTTP/127.5.22.20@KRBTEST.COM
I20260928 06:37:49.724920 18524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:49.713055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:40815 (local address 127.0.0.1:58478)
0928 06:37:49.715456 (+ 2401us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:49.715470 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:49.715558 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:49.716204 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:49.716207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:49.716224 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:49.716322 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:49.716328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:49.721726 (+ 5398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:49.721730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:49.722952 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:49.722963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:49.724332 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:49.724336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:49.724469 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:49.724510 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 06:37:49.724622 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 06:37:49.724744 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2341,"thread_start_us":77,"threads_started":1}
I20260928 06:37:49.726852 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:49.713592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:40511 (local address 127.0.0.1:38148)
0928 06:37:49.716513 (+ 2921us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:49.716525 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:49.716598 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:49.718389 (+ 1791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:49.718392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:49.718411 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:49.718503 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:49.718509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:49.719948 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:49.719952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:49.721364 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:49.721375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:49.726472 (+ 5097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:49.726476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:49.726575 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:49.726597 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 06:37:49.726656 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:37:49.726724 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2863,"thread_start_us":89,"threads_started":1}
I20260928 06:37:49.727365 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:49.714010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38148 (local address 127.5.22.1:40511)
0928 06:37:49.715308 (+ 1298us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:49.715314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:49.716684 (+ 1370us) server_negotiation.cc:407] Connection header received
0928 06:37:49.716760 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:49.716764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:49.716942 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:49.717067 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:49.718635 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:49.719804 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:49.725284 (+ 5480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:49.726131 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:49.726237 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:49.726408 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:49.727110 (+ 702us) server_negotiation.cc:299] Negotiation successful
0928 06:37:49.727220 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1200,"thread_start_us":116,"threads_started":1}
I20260928 06:37:49.728070 18522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:49.712790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:42785 (local address 127.0.0.1:32946)
0928 06:37:49.713483 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:49.713534 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:49.713671 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:49.714875 (+ 1204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:49.714890 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:49.714915 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:49.715052 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:49.715057 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:49.725313 (+ 10256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:49.725316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:49.726524 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:49.726537 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:49.727663 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:49.727666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:49.727759 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:49.727783 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 06:37:49.727840 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 06:37:49.727903 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":120,"threads_started":1}
I20260928 06:37:49.728693 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:49.712918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32946 (local address 127.5.22.2:42785)
0928 06:37:49.713222 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:49.713230 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:49.713815 (+ 585us) server_negotiation.cc:407] Connection header received
0928 06:37:49.714283 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:49.714290 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:49.714547 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:49.714696 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:49.715195 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:49.725165 (+ 9970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:49.726703 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:49.727509 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:49.728261 (+ 752us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:49.728412 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:49.728467 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 06:37:49.728558 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":92,"threads_started":1}
I20260928 06:37:49.728754 18526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:49.713672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58478 (local address 127.5.22.3:40815)
0928 06:37:49.714744 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:49.714751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:49.715654 (+ 903us) server_negotiation.cc:407] Connection header received
0928 06:37:49.715735 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:49.715740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:49.715942 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:49.716074 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:49.717287 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:49.721607 (+ 4320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:49.723146 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:49.724014 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:49.724119 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:49.724269 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:49.728503 (+ 4234us) server_negotiation.cc:299] Negotiation successful
0928 06:37:49.728618 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":961,"thread_start_us":79,"threads_started":1}
I20260928 06:37:49.730821 18158 tablet_service.cc:1467] Processing CreateTablet for tablet fa3faf39d5be4e4d95c6937c61f59814 (DEFAULT_TABLE table=test-table [id=a7b875f2ea654ee0b58e5c827ca8fb80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:49.731482 18282 tablet_service.cc:1467] Processing CreateTablet for tablet fa3faf39d5be4e4d95c6937c61f59814 (DEFAULT_TABLE table=test-table [id=a7b875f2ea654ee0b58e5c827ca8fb80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:49.731513 18429 tablet_service.cc:1467] Processing CreateTablet for tablet fa3faf39d5be4e4d95c6937c61f59814 (DEFAULT_TABLE table=test-table [id=a7b875f2ea654ee0b58e5c827ca8fb80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:49.732021 18158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa3faf39d5be4e4d95c6937c61f59814. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:49.732540 18282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa3faf39d5be4e4d95c6937c61f59814. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:49.732544 18429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa3faf39d5be4e4d95c6937c61f59814. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:49.746738 18529 tablet_bootstrap.cc:492] T fa3faf39d5be4e4d95c6937c61f59814 P cf1f77ab5c25419eb63d701ce814eb5c: Bootstrap starting.
I20260928 06:37:49.748281 18528 tablet_bootstrap.cc:492] T fa3faf39d5be4e4d95c6937c61f59814 P 6064a462f63a468b896673c4cf6629e7: Bootstrap starting.
I20260928 06:37:49.751892 18529 tablet_bootstrap.cc:654] T fa3faf39d5be4e4d95c6937c61f59814 P cf1f77ab5c25419eb63d701ce814eb5c: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:49.752827 18529 log.cc:824] T fa3faf39d5be4e4d95c6937c61f59814 P cf1f77ab5c25419eb63d701ce814eb5c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:49.753844 18528 tablet_bootstrap.cc:654] T fa3faf39d5be4e4d95c6937c61f59814 P 6064a462f63a468b896673c4cf6629e7: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:49.754745 18528 log.cc:824] T fa3faf39d5be4e4d95c6937c61f59814 P 6064a462f63a468b896673c4cf6629e7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:49.757020 18529 tablet_bootstrap.cc:492] T fa3faf39d5be4e4d95c6937c61f59814 P cf1f77ab5c25419eb63d701ce814eb5c: No bootstrap required, opened a new log
I20260928 06:37:49.757236 18529 ts_tablet_manager.cc:1397] T fa3faf39d5be4e4d95c6937c61f59814 P cf1f77ab5c25419eb63d701ce814eb5c: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260928 06:37:49.761865 18529 raft_consensus.cc:348] T fa3faf39d5be4e4d95c6937c61f59814 P cf1f77ab5c25419eb63d701ce814eb5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6064a462f63a468b896673c4cf6629e7" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 42785 } } peers { permanent_uuid: "ca05bc7ab5be4a2eaa6c0b8ddd561297" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40511 } } peers { permanent_uuid: "cf1f77ab5c25419eb63d701ce814eb5c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40815 } }
I20260928 06:37:49.762164 18529 raft_consensus.cc:374] T fa3faf39d5be4e4d95c6937c61f59814 P cf1f77ab5c25419eb63d701ce814eb5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:49.762212 18529 raft_consensus.cc:729] T fa3faf39d5be4e4d95c6937c61f59814 P cf1f77ab5c25419eb63d701ce814eb5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf1f77ab5c25419eb63d701ce814eb5c, State: Initialized, Role: FOLLOWER
I20260928 06:37:49.764220 18528 tablet_bootstrap.cc:492] T fa3faf39d5be4e4d95c6937c61f59814 P 6064a462f63a468b896673c4cf6629e7: No bootstrap required, opened a new log
I20260928 06:37:49.764441 18528 ts_tablet_manager.cc:1397] T fa3faf39d5be4e4d95c6937c61f59814 P 6064a462f63a468b896673c4cf6629e7: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20260928 06:37:49.768676 18528 raft_consensus.cc:348] T fa3faf39d5be4e4d95c6937c61f59814 P 6064a462f63a468b896673c4cf6629e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6064a462f63a468b896673c4cf6629e7" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 42785 } } peers { permanent_uuid: "ca05bc7ab5be4a2eaa6c0b8ddd561297" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40511 } } peers { permanent_uuid: "cf1f77ab5c25419eb63d701ce814eb5c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40815 } }
I20260928 06:37:49.768942 18528 raft_consensus.cc:374] T fa3faf39d5be4e4d95c6937c61f59814 P 6064a462f63a468b896673c4cf6629e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:49.768988 18528 raft_consensus.cc:729] T fa3faf39d5be4e4d95c6937c61f59814 P 6064a462f63a468b896673c4cf6629e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6064a462f63a468b896673c4cf6629e7, State: Initialized, Role: FOLLOWER
I20260928 06:37:49.769626 18530 tablet_bootstrap.cc:492] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297: Bootstrap starting.
I20260928 06:37:49.770643 18528 consensus_queue.cc:260] T fa3faf39d5be4e4d95c6937c61f59814 P 6064a462f63a468b896673c4cf6629e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6064a462f63a468b896673c4cf6629e7" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 42785 } } peers { permanent_uuid: "ca05bc7ab5be4a2eaa6c0b8ddd561297" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40511 } } peers { permanent_uuid: "cf1f77ab5c25419eb63d701ce814eb5c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40815 } }
I20260928 06:37:49.771739 18528 ts_tablet_manager.cc:1428] T fa3faf39d5be4e4d95c6937c61f59814 P 6064a462f63a468b896673c4cf6629e7: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 06:37:49.773763 18530 tablet_bootstrap.cc:654] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:49.774799 18529 consensus_queue.cc:260] T fa3faf39d5be4e4d95c6937c61f59814 P cf1f77ab5c25419eb63d701ce814eb5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6064a462f63a468b896673c4cf6629e7" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 42785 } } peers { permanent_uuid: "ca05bc7ab5be4a2eaa6c0b8ddd561297" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40511 } } peers { permanent_uuid: "cf1f77ab5c25419eb63d701ce814eb5c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40815 } }
I20260928 06:37:49.775816 18529 ts_tablet_manager.cc:1428] T fa3faf39d5be4e4d95c6937c61f59814 P cf1f77ab5c25419eb63d701ce814eb5c: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260928 06:37:49.776180 18361 heartbeater.cc:503] Master 127.5.22.62:34401 was elected leader, sending a full tablet report...
I20260928 06:37:49.780151 18495 heartbeater.cc:503] Master 127.5.22.62:34401 was elected leader, sending a full tablet report...
I20260928 06:37:49.782910 18530 log.cc:824] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:49.788458 18530 tablet_bootstrap.cc:492] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297: No bootstrap required, opened a new log
I20260928 06:37:49.788686 18530 ts_tablet_manager.cc:1397] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20260928 06:37:49.792948 18530 raft_consensus.cc:348] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6064a462f63a468b896673c4cf6629e7" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 42785 } } peers { permanent_uuid: "ca05bc7ab5be4a2eaa6c0b8ddd561297" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40511 } } peers { permanent_uuid: "cf1f77ab5c25419eb63d701ce814eb5c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40815 } }
I20260928 06:37:49.793251 18530 raft_consensus.cc:374] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:49.793303 18530 raft_consensus.cc:729] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca05bc7ab5be4a2eaa6c0b8ddd561297, State: Initialized, Role: FOLLOWER
I20260928 06:37:49.793867 18530 consensus_queue.cc:260] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6064a462f63a468b896673c4cf6629e7" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 42785 } } peers { permanent_uuid: "ca05bc7ab5be4a2eaa6c0b8ddd561297" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40511 } } peers { permanent_uuid: "cf1f77ab5c25419eb63d701ce814eb5c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40815 } }
I20260928 06:37:49.795116 18530 ts_tablet_manager.cc:1428] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 06:37:49.795578 18225 heartbeater.cc:503] Master 127.5.22.62:34401 was elected leader, sending a full tablet report...
W20260928 06:37:49.868876 18226 tablet.cc:2369] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:37:49.898540 18362 tablet.cc:2369] T fa3faf39d5be4e4d95c6937c61f59814 P 6064a462f63a468b896673c4cf6629e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:37:49.899260 18536 raft_consensus.cc:482] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:37:49.899505 18536 raft_consensus.cc:504] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6064a462f63a468b896673c4cf6629e7" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 42785 } } peers { permanent_uuid: "ca05bc7ab5be4a2eaa6c0b8ddd561297" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40511 } } peers { permanent_uuid: "cf1f77ab5c25419eb63d701ce814eb5c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40815 } }
I20260928 06:37:49.900995 18536 leader_election.cc:288] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6064a462f63a468b896673c4cf6629e7 (127.5.22.2:42785), cf1f77ab5c25419eb63d701ce814eb5c (127.5.22.3:40815)
I20260928 06:37:49.906694 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:49.901603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:57089 (local address 127.5.22.2:42785)
0928 06:37:49.901720 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:49.901725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:49.901974 (+ 249us) server_negotiation.cc:407] Connection header received
0928 06:37:49.902170 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:49.902173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:49.902221 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:49.902266 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:49.902752 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:49.903735 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:49.905323 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:49.905902 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:49.905986 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:49.906055 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:49.906434 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 06:37:49.906531 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:37:49.906881 18526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:49.901751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:50309 (local address 127.5.22.3:40815)
0928 06:37:49.901871 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:49.901876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:49.902232 (+ 356us) server_negotiation.cc:407] Connection header received
0928 06:37:49.902481 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:49.902485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:49.902538 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:49.902600 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:49.903110 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:49.903958 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:49.905484 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:49.906124 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:49.906212 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:49.906275 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:49.906670 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 06:37:49.906758 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:37:49.907346 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:49.901355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:40815 (local address 127.5.22.1:50309)
0928 06:37:49.902134 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:49.902148 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:49.902341 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:49.902742 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:49.902745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:49.902761 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:49.902886 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:49.902938 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:49.904130 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:49.904133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:49.905335 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:49.905346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:49.906413 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:49.906417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:49.906519 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:49.906543 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 06:37:49.907102 (+ 559us) client_negotiation.cc:240] Negotiation successful
0928 06:37:49.907182 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":119,"threads_started":1}
I20260928 06:37:49.907572 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:49.901218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:42785 (local address 127.5.22.1:57089)
0928 06:37:49.901885 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:49.901899 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:49.902052 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:49.902448 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:49.902450 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:49.902470 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:49.902615 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:49.902621 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:49.903886 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:49.903890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:49.905179 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:49.905190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:49.906152 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:49.906155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:49.906255 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:49.906277 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 06:37:49.907262 (+ 985us) client_negotiation.cc:240] Negotiation successful
0928 06:37:49.907365 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":8,"thread_start_us":116,"threads_started":1}
I20260928 06:37:49.908208 18449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa3faf39d5be4e4d95c6937c61f59814" candidate_uuid: "ca05bc7ab5be4a2eaa6c0b8ddd561297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1f77ab5c25419eb63d701ce814eb5c" is_pre_election: true
I20260928 06:37:49.908288 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa3faf39d5be4e4d95c6937c61f59814" candidate_uuid: "ca05bc7ab5be4a2eaa6c0b8ddd561297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6064a462f63a468b896673c4cf6629e7" is_pre_election: true
I20260928 06:37:49.908551 18449 raft_consensus.cc:2454] T fa3faf39d5be4e4d95c6937c61f59814 P cf1f77ab5c25419eb63d701ce814eb5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca05bc7ab5be4a2eaa6c0b8ddd561297 in term 0.
I20260928 06:37:49.908591 18313 raft_consensus.cc:2454] T fa3faf39d5be4e4d95c6937c61f59814 P 6064a462f63a468b896673c4cf6629e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca05bc7ab5be4a2eaa6c0b8ddd561297 in term 0.
I20260928 06:37:49.909101 18112 leader_election.cc:302] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6064a462f63a468b896673c4cf6629e7, ca05bc7ab5be4a2eaa6c0b8ddd561297, cf1f77ab5c25419eb63d701ce814eb5c; no voters:
I20260928 06:37:49.909503 18536 raft_consensus.cc:2781] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:37:49.909603 18536 raft_consensus.cc:482] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:37:49.909647 18536 raft_consensus.cc:3037] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:49.911092 18536 raft_consensus.cc:504] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6064a462f63a468b896673c4cf6629e7" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 42785 } } peers { permanent_uuid: "ca05bc7ab5be4a2eaa6c0b8ddd561297" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40511 } } peers { permanent_uuid: "cf1f77ab5c25419eb63d701ce814eb5c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40815 } }
I20260928 06:37:49.911522 18536 leader_election.cc:288] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297 [CANDIDATE]: Term 1 election: Requested vote from peers 6064a462f63a468b896673c4cf6629e7 (127.5.22.2:42785), cf1f77ab5c25419eb63d701ce814eb5c (127.5.22.3:40815)
I20260928 06:37:49.911835 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa3faf39d5be4e4d95c6937c61f59814" candidate_uuid: "ca05bc7ab5be4a2eaa6c0b8ddd561297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6064a462f63a468b896673c4cf6629e7"
I20260928 06:37:49.911955 18313 raft_consensus.cc:3037] T fa3faf39d5be4e4d95c6937c61f59814 P 6064a462f63a468b896673c4cf6629e7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:49.912014 18449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa3faf39d5be4e4d95c6937c61f59814" candidate_uuid: "ca05bc7ab5be4a2eaa6c0b8ddd561297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1f77ab5c25419eb63d701ce814eb5c"
I20260928 06:37:49.912134 18449 raft_consensus.cc:3037] T fa3faf39d5be4e4d95c6937c61f59814 P cf1f77ab5c25419eb63d701ce814eb5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:49.913163 18313 raft_consensus.cc:2454] T fa3faf39d5be4e4d95c6937c61f59814 P 6064a462f63a468b896673c4cf6629e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca05bc7ab5be4a2eaa6c0b8ddd561297 in term 1.
I20260928 06:37:49.913424 18449 raft_consensus.cc:2454] T fa3faf39d5be4e4d95c6937c61f59814 P cf1f77ab5c25419eb63d701ce814eb5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca05bc7ab5be4a2eaa6c0b8ddd561297 in term 1.
I20260928 06:37:49.913520 18112 leader_election.cc:302] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6064a462f63a468b896673c4cf6629e7, ca05bc7ab5be4a2eaa6c0b8ddd561297; no voters:
I20260928 06:37:49.913815 18536 raft_consensus.cc:2781] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:49.914072 18536 raft_consensus.cc:686] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297 [term 1 LEADER]: Becoming Leader. State: Replica: ca05bc7ab5be4a2eaa6c0b8ddd561297, State: Running, Role: LEADER
I20260928 06:37:49.914454 18536 consensus_queue.cc:237] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6064a462f63a468b896673c4cf6629e7" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 42785 } } peers { permanent_uuid: "ca05bc7ab5be4a2eaa6c0b8ddd561297" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40511 } } peers { permanent_uuid: "cf1f77ab5c25419eb63d701ce814eb5c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40815 } }
I20260928 06:37:49.917992 18025 catalog_manager.cc:5495] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca05bc7ab5be4a2eaa6c0b8ddd561297 (127.5.22.1). New cstate: current_term: 1 leader_uuid: "ca05bc7ab5be4a2eaa6c0b8ddd561297" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6064a462f63a468b896673c4cf6629e7" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 42785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca05bc7ab5be4a2eaa6c0b8ddd561297" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf1f77ab5c25419eb63d701ce814eb5c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40815 } health_report { overall_health: UNKNOWN } } }
W20260928 06:37:49.921406 18496 tablet.cc:2369] T fa3faf39d5be4e4d95c6937c61f59814 P cf1f77ab5c25419eb63d701ce814eb5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:37:49.977859 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:49.972956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38156 (local address 127.5.22.1:40511)
0928 06:37:49.973078 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:49.973085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:49.973104 (+ 19us) server_negotiation.cc:407] Connection header received
0928 06:37:49.973193 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:49.973195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:49.973251 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:49.973388 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:37:49.973675 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:49.974209 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:49.975538 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:49.975835 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:49.975948 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:49.976436 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:37:49.976985 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:37:49.977135 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:49.977612 (+ 477us) server_negotiation.cc:299] Negotiation successful
0928 06:37:49.977686 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:37:49.984966 18449 raft_consensus.cc:1261] T fa3faf39d5be4e4d95c6937c61f59814 P cf1f77ab5c25419eb63d701ce814eb5c [term 1 FOLLOWER]: Refusing update from remote peer ca05bc7ab5be4a2eaa6c0b8ddd561297: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:49.985113 18313 raft_consensus.cc:1261] T fa3faf39d5be4e4d95c6937c61f59814 P 6064a462f63a468b896673c4cf6629e7 [term 1 FOLLOWER]: Refusing update from remote peer ca05bc7ab5be4a2eaa6c0b8ddd561297: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:49.985729 18536 consensus_queue.cc:1035] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6064a462f63a468b896673c4cf6629e7" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 42785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:49.985958 18539 consensus_queue.cc:1035] T fa3faf39d5be4e4d95c6937c61f59814 P ca05bc7ab5be4a2eaa6c0b8ddd561297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf1f77ab5c25419eb63d701ce814eb5c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:50.002492 18544 mvcc.cc:205] Tried to move back new op lower bound from 7334205317050118144 to 7334205316772782080. Current Snapshot: MvccSnapshot[applied={T|T < 7334205317050118144}]
I20260928 06:37:50.009291 18025 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:37:50.009480 18025 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:37:50.013845 18025 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5090a098891a48a1a23a7b88fd6f7a1f: Sending DeleteTablet for 3 replicas of tablet fa3faf39d5be4e4d95c6937c61f59814
I20260928 06:37:50.015137 18282 tablet_service.cc:1514] Processing DeleteTablet for tablet fa3faf39d5be4e4d95c6937c61f59814 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:50 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:32946
I20260928 06:37:50.015700 18158 tablet_service.cc:1514] Processing DeleteTablet for tablet fa3faf39d5be4e4d95c6937c61f59814 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:50 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:38148
I20260928 06:37:50.016683 18429 tablet_service.cc:1514] Processing DeleteTablet for tablet fa3faf39d5be4e4d95c6937c61f59814 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:50 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:58478
I20260928 06:37:50.018990 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 18063
I20260928 06:37:50.021577 18543 mvcc.cc:205] Tried to move back new op lower bound from 7334205317050118144 to 7334205316772782080. Current Snapshot: MvccSnapshot[applied={T|T < 7334205317050118144 or (T in {7334205317050118144})}]
I20260928 06:37:50.021874 18554 tablet_replica.cc:330] T fa3faf39d5be4e4d95c6937c61f59814 P cf1f77ab5c25419eb63d701ce814eb5c: stopping tablet replica
I20260928 06:37:50.022169 18552 tablet_replica.cc:330] T fa3faf39d5be4e4d95c6937c61f59814 P 6064a462f63a468b896673c4cf6629e7: stopping tablet replica
I20260928 06:37:50.022364 18554 raft_consensus.cc:2229] T fa3faf39d5be4e4d95c6937c61f59814 P cf1f77ab5c25419eb63d701ce814eb5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:37:50.022573 18552 raft_consensus.cc:2229] T fa3faf39d5be4e4d95c6937c61f59814 P 6064a462f63a468b896673c4cf6629e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:37:50.022809 18554 raft_consensus.cc:2258] T fa3faf39d5be4e4d95c6937c61f59814 P cf1f77ab5c25419eb63d701ce814eb5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:50.022984 18552 raft_consensus.cc:2258] T fa3faf39d5be4e4d95c6937c61f59814 P 6064a462f63a468b896673c4cf6629e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:50.024466 18554 ts_tablet_manager.cc:1905] T fa3faf39d5be4e4d95c6937c61f59814 P cf1f77ab5c25419eb63d701ce814eb5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:37:50.024602 18552 ts_tablet_manager.cc:1905] T fa3faf39d5be4e4d95c6937c61f59814 P 6064a462f63a468b896673c4cf6629e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:37:50.027773 18552 ts_tablet_manager.cc:1918] T fa3faf39d5be4e4d95c6937c61f59814 P 6064a462f63a468b896673c4cf6629e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:37:50.027969 18552 log.cc:1196] T fa3faf39d5be4e4d95c6937c61f59814 P 6064a462f63a468b896673c4cf6629e7: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-1/wal/wals/fa3faf39d5be4e4d95c6937c61f59814
W20260928 06:37:50.028155 18011 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.22.1:40511 (error 108)
I20260928 06:37:50.028468 18552 ts_tablet_manager.cc:1939] T fa3faf39d5be4e4d95c6937c61f59814 P 6064a462f63a468b896673c4cf6629e7: Deleting consensus metadata
I20260928 06:37:50.029814 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.029469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:40511 (local address 127.0.0.1:38168)
0928 06:37:50.029614 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.029709 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.1:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 06:37:50.030076 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 18229
W20260928 06:37:50.030697 18011 catalog_manager.cc:4584] TS ca05bc7ab5be4a2eaa6c0b8ddd561297 (127.5.22.1:40511): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fa3faf39d5be4e4d95c6937c61f59814: Network error: Client connection negotiation failed: client connection to 127.5.22.1:40511: connect: Connection refused (error 111)
I20260928 06:37:50.032483 18554 ts_tablet_manager.cc:1918] T fa3faf39d5be4e4d95c6937c61f59814 P cf1f77ab5c25419eb63d701ce814eb5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:37:50.032631 18554 log.cc:1196] T fa3faf39d5be4e4d95c6937c61f59814 P cf1f77ab5c25419eb63d701ce814eb5c: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/fa3faf39d5be4e4d95c6937c61f59814
I20260928 06:37:50.032994 18554 ts_tablet_manager.cc:1939] T fa3faf39d5be4e4d95c6937c61f59814 P cf1f77ab5c25419eb63d701ce814eb5c: Deleting consensus metadata
I20260928 06:37:50.034296 18009 catalog_manager.cc:4841] TS cf1f77ab5c25419eb63d701ce814eb5c (127.5.22.3:40815): tablet fa3faf39d5be4e4d95c6937c61f59814 (table test-table [id=a7b875f2ea654ee0b58e5c827ca8fb80]) successfully deleted
I20260928 06:37:50.037949 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.037570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:42785 (local address 127.0.0.1:32960)
0928 06:37:50.037803 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.037861 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.2:42785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260928 06:37:50.038183 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 18363
I20260928 06:37:50.045837 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 17992
I20260928 06:37:50.055253 5208 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 06:37:50.390059 5208 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 06:37:50.390159 5208 mini_ranger.cc:66] Stopping Ranger...
I20260928 06:37:50.734954 5208 mini_ranger.cc:68] Stopped Ranger
2026-09-28 06:37:50.735 UTC [17059] LOG: received smart shutdown request
2026-09-28 06:37:50.740 UTC [17059] LOG: background worker "logical replication launcher" (PID 17068) exited with exit code 1
2026-09-28 06:37:50.740 UTC [17063] LOG: shutting down
2026-09-28 06:37:50.932 UTC [17059] LOG: database system is shut down
2026-09-28T06:37:50Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48596 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:51 dist-test-slave-j0qf krb5kdc[18565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:51 dist-test-slave-j0qf krb5kdc[18565](info): set up 2 sockets
Sep 28 06:37:51 dist-test-slave-j0qf krb5kdc[18565](info): commencing operation
krb5kdc: starting...
W20260928 06:37:52.105288 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:37:52 dist-test-slave-j0qf krb5kdc[18565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:37:52.233246 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:43013
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:38829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:43013,127.5.22.61:38605,127.5.22.60:32967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:52.347748 18578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:52.348105 18578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:52.348176 18578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:52.352344 18578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:52.352468 18578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:52.352502 18578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:52.352545 18578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:52.352581 18578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:52.358377 18578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:38829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:43013,127.5.22.61:38605,127.5.22.60:32967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:43013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:52.359558 18578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:52.360625 18578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:52.369241 18583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:52.369241 18584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:52.369261 18586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:52.369496 18578 server_base.cc:1034] running on GCE node
I20260928 06:37:52.370121 18578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:52.371272 18578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:52.372445 18578 hybrid_clock.cc:648] HybridClock initialized: now 1790577472372416 us; error 43 us; skew 500 ppm
Sep 28 06:37:52 dist-test-slave-j0qf krb5kdc[18565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577472, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:52.375300 18578 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:37:52.376147 18578 webserver.cc:460] Webserver started at http://127.5.22.62:45159/ using document root <none> and password file <none>
I20260928 06:37:52.376721 18578 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:52.376840 18578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:52.377068 18578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:52.379096 18578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4"
format_stamp: "Formatted at 2026-09-28 06:37:52 on dist-test-slave-j0qf"
server_key: "e25c00f57badded132b02f0812250e82"
server_key_iv: "f1bfe805111fab16afc647b658f29c8c"
server_key_version: "encryptionkey@0"
I20260928 06:37:52.379654 18578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4"
format_stamp: "Formatted at 2026-09-28 06:37:52 on dist-test-slave-j0qf"
server_key: "e25c00f57badded132b02f0812250e82"
server_key_iv: "f1bfe805111fab16afc647b658f29c8c"
server_key_version: "encryptionkey@0"
I20260928 06:37:52.383488 18578 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 06:37:52.386122 18593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:52.387518 18578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:37:52.387672 18578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4"
format_stamp: "Formatted at 2026-09-28 06:37:52 on dist-test-slave-j0qf"
server_key: "e25c00f57badded132b02f0812250e82"
server_key_iv: "f1bfe805111fab16afc647b658f29c8c"
server_key_version: "encryptionkey@0"
I20260928 06:37:52.387789 18578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:52.413426 18578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:52.416937 18578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:52.417145 18578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:52.424480 18578 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:43013
I20260928 06:37:52.424508 18645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:43013 every 8 connection(s)
I20260928 06:37:52.425417 18578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:52.428553 18646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:52.430500 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 18578
I20260928 06:37:52.430642 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:37:52.430939 5208 external_mini_cluster.cc:1428] Setting key c8762adf5187f4fb189a0522380f24a8
I20260928 06:37:52.432845 18646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43013 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.22.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.61@KRBTEST.COM" created.
Sep 28 06:37:52 dist-test-slave-j0qf krb5kdc[18565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577472, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:37:52.451027 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.436415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38138 (local address 127.5.22.62:43013)
0928 06:37:52.436802 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:52.436817 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:52.436874 (+ 57us) server_negotiation.cc:407] Connection header received
0928 06:37:52.437379 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:52.437386 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:52.437671 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:52.437904 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:52.438888 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.440525 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.441587 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.442014 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.442154 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:52.445451 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:52.445457 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:52.445468 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:52.445492 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:52.447730 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.448346 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.448351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.448352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.448507 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.448931 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.448934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.448936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.449312 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:52.449436 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:52.449914 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 06:37:52.450164 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":234,"thread_start_us":98,"threads_started":1}
I20260928 06:37:52.451027 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.435480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43013 (local address 127.0.0.1:38138)
0928 06:37:52.435997 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.436051 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:52.437035 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:52.437934 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:52.437938 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:52.438436 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:52.438703 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:52.438717 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.440661 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.440664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:52.441447 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:52.441457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.442165 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.442168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:52.442405 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:52.443035 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:52.443051 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:52.445153 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:52.447873 (+ 2720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:52.447886 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:52.447899 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:52.448219 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:52.448618 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:52.448624 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:52.448627 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:52.448792 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:52.449441 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:52.449450 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:52.449642 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 06:37:52.449857 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 06:37:52.450280 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":91,"thread_start_us":164,"threads_started":1}
I20260928 06:37:52.453732 18646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:37:52.454607 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.454227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:38605 (local address 127.0.0.1:39398)
0928 06:37:52.454415 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.454525 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.61:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 06:37:52.454852 18596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.22.61:38605: connect: Connection refused (error 111)
I20260928 06:37:52.455756 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.455567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:38605 (local address 127.0.0.1:39408)
0928 06:37:52.455663 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.455707 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.61:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 06:37:52.455967 18646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.61:38605: Network error: Client connection negotiation failed: client connection to 127.5.22.61:38605: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.61.keytab.
Entry for principal kudu/127.5.22.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.61.keytab.
I20260928 06:37:52.461643 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.61:38605
--webserver_interface=127.5.22.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.61.keytab
--principal=kudu/127.5.22.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:38829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:43013,127.5.22.61:38605,127.5.22.60:32967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:37:52.505221 18646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 } attempt: 1
I20260928 06:37:52.506186 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.505832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:38605 (local address 127.0.0.1:39420)
0928 06:37:52.506010 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.506089 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.61:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 06:37:52.506995 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.506743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:38605 (local address 127.0.0.1:39430)
0928 06:37:52.506861 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.506919 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.61:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 06:37:52.507155 18646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.61:38605: Network error: Client connection negotiation failed: client connection to 127.5.22.61:38605: connect: Connection refused (error 111)
W20260928 06:37:52.574735 18653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:52.575059 18653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:52.575145 18653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 06:37:52.575325 18646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 } attempt: 2
I20260928 06:37:52.576212 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.575871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:38605 (local address 127.0.0.1:39440)
0928 06:37:52.576045 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.576118 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.61:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 06:37:52.577037 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.576791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:38605 (local address 127.0.0.1:39456)
0928 06:37:52.576915 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.576970 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.61:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 06:37:52.577189 18646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.61:38605: Network error: Client connection negotiation failed: client connection to 127.5.22.61:38605: connect: Connection refused (error 111)
W20260928 06:37:52.579900 18653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:52.579954 18653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:52.579973 18653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:52.580009 18653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:52.580041 18653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:52.585588 18653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:38829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:43013,127.5.22.61:38605,127.5.22.60:32967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.61.keytab
--principal=kudu/127.5.22.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.61:38605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.22.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:52.586731 18653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:52.587764 18653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:52.596164 18658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:52.596236 18661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:52.596161 18659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:52.596468 18653 server_base.cc:1034] running on GCE node
I20260928 06:37:52.596995 18653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:52.597980 18653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:52.599185 18653 hybrid_clock.cc:648] HybridClock initialized: now 1790577472599172 us; error 24 us; skew 500 ppm
Sep 28 06:37:52 dist-test-slave-j0qf krb5kdc[18565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577472, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:52.601944 18653 init.cc:377] Logged in from keytab as kudu/127.5.22.61@KRBTEST.COM (short username kudu)
I20260928 06:37:52.602897 18653 webserver.cc:460] Webserver started at http://127.5.22.61:46763/ using document root <none> and password file <none>
I20260928 06:37:52.603475 18653 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:52.603540 18653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:52.603780 18653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:52.605777 18653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-1/data/instance:
uuid: "09291973153a4bd0ba2786dfb7c6ad7a"
format_stamp: "Formatted at 2026-09-28 06:37:52 on dist-test-slave-j0qf"
server_key: "690d1d9373a3bfc4c6cb81f520c95e33"
server_key_iv: "c611c55b4b83d09bdbc4d719793c7b40"
server_key_version: "encryptionkey@0"
I20260928 06:37:52.606500 18653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-1/wal/instance:
uuid: "09291973153a4bd0ba2786dfb7c6ad7a"
format_stamp: "Formatted at 2026-09-28 06:37:52 on dist-test-slave-j0qf"
server_key: "690d1d9373a3bfc4c6cb81f520c95e33"
server_key_iv: "c611c55b4b83d09bdbc4d719793c7b40"
server_key_version: "encryptionkey@0"
I20260928 06:37:52.610291 18653 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 06:37:52.612749 18668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:52.613998 18653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:37:52.614100 18653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-1/wal
uuid: "09291973153a4bd0ba2786dfb7c6ad7a"
format_stamp: "Formatted at 2026-09-28 06:37:52 on dist-test-slave-j0qf"
server_key: "690d1d9373a3bfc4c6cb81f520c95e33"
server_key_iv: "c611c55b4b83d09bdbc4d719793c7b40"
server_key_version: "encryptionkey@0"
I20260928 06:37:52.614240 18653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:52.626789 18653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:52.630015 18653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:52.630385 18653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:52.637526 18653 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.61:38605
I20260928 06:37:52.637553 18721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.61:38605 every 8 connection(s)
I20260928 06:37:52.638532 18653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-1/data/info.pb
I20260928 06:37:52.641359 18722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:52.645200 18722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43013 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:37:52.648029 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 18653
I20260928 06:37:52.648157 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-1/wal/instance
I20260928 06:37:52.648443 5208 external_mini_cluster.cc:1428] Setting key 432737b9598995eeece1abdf0ae37419
Sep 28 06:37:52 dist-test-slave-j0qf krb5kdc[18565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577472, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.61@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.22.60@KRBTEST.COM; defaulting to no policy
I20260928 06:37:52.660415 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.647185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.5.22.62:43013)
0928 06:37:52.647350 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:52.647355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:52.648383 (+ 1028us) server_negotiation.cc:407] Connection header received
0928 06:37:52.649394 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:52.649397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:52.649459 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:52.649569 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:52.650905 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.651448 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.653295 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.653567 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.653657 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:52.656668 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:52.656671 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:52.656673 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:52.656697 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:52.658246 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.658815 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.658824 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.658825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.658902 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.659254 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.659258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.659260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.659458 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:52.659566 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:52.660105 (+ 539us) server_negotiation.cc:299] Negotiation successful
0928 06:37:52.660236 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.60@KRBTEST.COM" created.
I20260928 06:37:52.661492 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.647591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43013 (local address 127.0.0.1:38154)
0928 06:37:52.648191 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.648255 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:52.649171 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:52.649747 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:52.649753 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:52.650134 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:52.650734 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:52.650748 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.651585 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.651592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:52.653135 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:52.653152 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.653745 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.653750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:52.654000 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:52.654696 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:52.654726 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:52.656487 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:52.658397 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:52.658403 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:52.658422 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:52.658699 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:52.658996 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:52.659000 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:52.659002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:52.659155 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:52.659568 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:52.659577 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:52.659845 (+ 268us) client_negotiation.cc:769] Sending connection context
0928 06:37:52.660046 (+ 201us) client_negotiation.cc:240] Negotiation successful
0928 06:37:52.660392 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":187,"threads_started":1}
I20260928 06:37:52.662997 18722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:37:52.668355 18646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 } attempt: 3
Sep 28 06:37:52 dist-test-slave-j0qf krb5kdc[18565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577472, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.61@KRBTEST.COM for kudu/127.5.22.61@KRBTEST.COM
I20260928 06:37:52.673151 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.663523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:38605 (local address 127.0.0.1:39466)
0928 06:37:52.663646 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.663660 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:52.663746 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:52.664409 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:52.664412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:52.664589 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:52.664673 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:52.664678 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.665644 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.665648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:52.666199 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:52.666207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.666856 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.666859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:52.666967 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:52.667470 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:52.667484 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:52.669077 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:52.671286 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:52.671290 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:52.671292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:52.671564 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:52.671903 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:52.671906 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:52.671908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:52.672005 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:52.672577 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:52.672581 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:52.672753 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 06:37:52.672837 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 06:37:52.673012 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":43}
I20260928 06:37:52.673693 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.663670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39466 (local address 127.5.22.61:38605)
0928 06:37:52.663984 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:52.663992 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:52.664010 (+ 18us) server_negotiation.cc:407] Connection header received
0928 06:37:52.664042 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:52.664046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:52.664191 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:52.664440 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:52.664779 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.665530 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.666340 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.666741 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.666857 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:52.669219 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:52.669232 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:52.669251 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:52.669274 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:52.671155 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.671700 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.671708 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.671710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.671808 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.672148 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.672151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.672153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.672461 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:52.672609 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:52.672967 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 06:37:52.673131 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":115,"threads_started":1}
Sep 28 06:37:52 dist-test-slave-j0qf krb5kdc[18565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577472, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for kudu/127.5.22.61@KRBTEST.COM
I20260928 06:37:52.675908 18722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.60.keytab.
Entry for principal kudu/127.5.22.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.60.keytab.
I20260928 06:37:52.676797 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.676466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:32967 (local address 127.0.0.1:42240)
0928 06:37:52.676610 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.676700 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 06:37:52.677003 18672 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.22.60:32967: connect: Connection refused (error 111)
I20260928 06:37:52.677960 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.677705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:32967 (local address 127.0.0.1:42252)
0928 06:37:52.677823 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.677882 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 06:37:52.678130 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.60:32967
--webserver_interface=127.5.22.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.60.keytab
--principal=kudu/127.5.22.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:38829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:43013,127.5.22.61:38605,127.5.22.60:32967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:52.678476 18722 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.60:32967: Network error: Client connection negotiation failed: client connection to 127.5.22.60:32967: connect: Connection refused (error 111)
I20260928 06:37:52.679883 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.668855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:38605 (local address 127.0.0.1:39474)
0928 06:37:52.669021 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.669037 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:52.669124 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:52.669858 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:52.669861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:52.670137 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:52.670798 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:52.670827 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.671717 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.671720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:52.672484 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:52.672495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.673313 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.673317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:52.673496 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:52.674068 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:52.674083 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:52.675732 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:52.677929 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:52.677933 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:52.677935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:52.678202 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:52.678839 (+ 637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:52.678845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:52.678849 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:52.678962 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:52.679327 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:52.679332 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:52.679474 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 06:37:52.679606 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 06:37:52.679766 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 06:37:52.680068 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.669253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39474 (local address 127.5.22.61:38605)
0928 06:37:52.669629 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:52.669636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:52.669648 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:37:52.669676 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:52.669678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:52.669710 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:52.669831 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:52.670961 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.671587 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.672631 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.673155 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.673822 (+ 667us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:52.675885 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:52.675888 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:52.675930 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:52.675958 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:52.677807 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.678648 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.678652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.678654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.678741 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.679064 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.679066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.679067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.679235 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:52.679309 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:52.679691 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 06:37:52.679833 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":313,"thread_start_us":145,"threads_started":1}
I20260928 06:37:52.680889 18646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:37:52.681638 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.681321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:32967 (local address 127.0.0.1:42254)
0928 06:37:52.681482 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.681548 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 06:37:52.682436 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.682070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:32967 (local address 127.0.0.1:42262)
0928 06:37:52.682241 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.682302 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260928 06:37:52.682578 18646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.60:32967: Network error: Client connection negotiation failed: client connection to 127.5.22.60:32967: connect: Connection refused (error 111)
I20260928 06:37:52.713765 18646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } attempt: 1
I20260928 06:37:52.714794 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.714439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:32967 (local address 127.0.0.1:42276)
0928 06:37:52.714616 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.714694 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 06:37:52.715728 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.715489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:32967 (local address 127.0.0.1:42282)
0928 06:37:52.715603 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.715659 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 06:37:52.715997 18646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.60:32967: Network error: Client connection negotiation failed: client connection to 127.5.22.60:32967: connect: Connection refused (error 111)
I20260928 06:37:52.727741 18722 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } attempt: 1
I20260928 06:37:52.728646 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.728343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:32967 (local address 127.0.0.1:42298)
0928 06:37:52.728504 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.728568 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 06:37:52.729391 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.729182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:32967 (local address 127.0.0.1:42302)
0928 06:37:52.729291 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.729342 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 06:37:52.729535 18722 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.60:32967: Network error: Client connection negotiation failed: client connection to 127.5.22.60:32967: connect: Connection refused (error 111)
I20260928 06:37:52.791247 18646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } attempt: 2
I20260928 06:37:52.792191 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.791835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:32967 (local address 127.0.0.1:42312)
0928 06:37:52.792033 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.792095 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 06:37:52.793406 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.793205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:32967 (local address 127.0.0.1:42324)
0928 06:37:52.793306 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.793350 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 06:37:52.793570 18646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.60:32967: Network error: Client connection negotiation failed: client connection to 127.5.22.60:32967: connect: Connection refused (error 111)
I20260928 06:37:52.797705 18722 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } attempt: 2
W20260928 06:37:52.797683 18730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:52.798053 18730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:52.798164 18730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 06:37:52.798524 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.798188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:32967 (local address 127.0.0.1:42328)
0928 06:37:52.798352 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.798455 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 06:37:52.799177 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.798981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:32967 (local address 127.0.0.1:42344)
0928 06:37:52.799086 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.799134 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 06:37:52.799296 18722 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.60:32967: Network error: Client connection negotiation failed: client connection to 127.5.22.60:32967: connect: Connection refused (error 111)
W20260928 06:37:52.803028 18730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:52.803095 18730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:52.803160 18730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:52.803203 18730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:52.803222 18730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:52.808944 18730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:38829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:43013,127.5.22.61:38605,127.5.22.60:32967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.60.keytab
--principal=kudu/127.5.22.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.60:32967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.22.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:52.810084 18730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:52.811173 18730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:52.820286 18735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:52.820277 18738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:52.820279 18736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:52.820631 18730 server_base.cc:1034] running on GCE node
I20260928 06:37:52.821164 18730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:52.822274 18730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:52.823503 18730 hybrid_clock.cc:648] HybridClock initialized: now 1790577472823468 us; error 47 us; skew 500 ppm
Sep 28 06:37:52 dist-test-slave-j0qf krb5kdc[18565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577472, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:52.826517 18730 init.cc:377] Logged in from keytab as kudu/127.5.22.60@KRBTEST.COM (short username kudu)
I20260928 06:37:52.827414 18730 webserver.cc:460] Webserver started at http://127.5.22.60:44399/ using document root <none> and password file <none>
I20260928 06:37:52.827939 18730 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:52.827998 18730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:52.828166 18730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:52.830206 18730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-2/data/instance:
uuid: "2c35be34ef5345b69c150659f903a0fd"
format_stamp: "Formatted at 2026-09-28 06:37:52 on dist-test-slave-j0qf"
server_key: "57e145e5b314a2592ae14649ca7f6c61"
server_key_iv: "4534df56e735217fbe284320fe0e2c1a"
server_key_version: "encryptionkey@0"
I20260928 06:37:52.830791 18730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-2/wal/instance:
uuid: "2c35be34ef5345b69c150659f903a0fd"
format_stamp: "Formatted at 2026-09-28 06:37:52 on dist-test-slave-j0qf"
server_key: "57e145e5b314a2592ae14649ca7f6c61"
server_key_iv: "4534df56e735217fbe284320fe0e2c1a"
server_key_version: "encryptionkey@0"
I20260928 06:37:52.834532 18730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 06:37:52.837101 18745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:52.838415 18730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:37:52.838550 18730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-2/wal
uuid: "2c35be34ef5345b69c150659f903a0fd"
format_stamp: "Formatted at 2026-09-28 06:37:52 on dist-test-slave-j0qf"
server_key: "57e145e5b314a2592ae14649ca7f6c61"
server_key_iv: "4534df56e735217fbe284320fe0e2c1a"
server_key_version: "encryptionkey@0"
I20260928 06:37:52.838663 18730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:52.882107 18730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:52.885850 18730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:52.886098 18730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:52.890468 18722 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } attempt: 3
I20260928 06:37:52.891314 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.891023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:32967 (local address 127.0.0.1:42358)
0928 06:37:52.891148 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.891219 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 06:37:52.892180 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.891870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:32967 (local address 127.0.0.1:42362)
0928 06:37:52.892032 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.892102 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260928 06:37:52.892444 18722 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.60:32967: Network error: Client connection negotiation failed: client connection to 127.5.22.60:32967: connect: Connection refused (error 111)
I20260928 06:37:52.892745 18646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } attempt: 3
I20260928 06:37:52.893451 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.893199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:32967 (local address 127.0.0.1:42366)
0928 06:37:52.893330 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.893381 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 06:37:52.894089 18730 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.60:32967
I20260928 06:37:52.894150 18797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.60:32967 every 8 connection(s)
I20260928 06:37:52.894208 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.893953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:32967 (local address 127.0.0.1:42370)
0928 06:37:52.894082 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.894141 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 06:37:52.894364 18646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.60:32967: Network error: Client connection negotiation failed: client connection to 127.5.22.60:32967: connect: Connection refused (error 111)
I20260928 06:37:52.895195 18730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-2/data/info.pb
I20260928 06:37:52.896386 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 18730
I20260928 06:37:52.896524 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577308481727-5208-0/minicluster-data/master-2/wal/instance
I20260928 06:37:52.896828 5208 external_mini_cluster.cc:1428] Setting key 7dcb6fcf993e887300cb6c63e055464b
I20260928 06:37:52.898533 18798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:52.902871 18798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43013 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:37:52 dist-test-slave-j0qf krb5kdc[18565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.60@KRBTEST.COM
Sep 28 06:37:52 dist-test-slave-j0qf krb5kdc[18565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577472, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.60@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:37:52.911795 18801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.898183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42374 (local address 127.5.22.60:32967)
0928 06:37:52.898650 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:52.898670 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:52.898712 (+ 42us) server_negotiation.cc:407] Connection header received
0928 06:37:52.899340 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:52.899355 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:52.899672 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:52.899954 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:52.900875 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.901852 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.903035 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.903616 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.903739 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:52.906654 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:52.906660 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:52.906669 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:52.906695 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:52.908509 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.908977 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.908990 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.908995 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.909105 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.909427 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.909430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.909432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.909950 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:52.910193 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:52.910485 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 06:37:52.910792 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":206,"threads_started":1}
I20260928 06:37:52.915012 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.904967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38166 (local address 127.5.22.62:43013)
0928 06:37:52.905123 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:52.905128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:52.905473 (+ 345us) server_negotiation.cc:407] Connection header received
0928 06:37:52.905643 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:52.905646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:52.905699 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:52.905789 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:52.906568 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.907193 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.908218 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.908449 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.908560 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:52.910762 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:52.910766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:52.910770 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:52.910799 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:52.912414 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.913541 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.913548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.913552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.913645 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.913960 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.913964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.913966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.914239 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:52.914383 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:52.914739 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 06:37:52.914869 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 06:37:52.915096 18802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.904872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43013 (local address 127.0.0.1:38166)
0928 06:37:52.905317 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.905386 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:52.905533 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:52.905839 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:52.905844 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:52.906248 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:52.906448 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:52.906455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.907308 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.907311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:52.908091 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:52.908100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.908563 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.908567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:52.908678 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:52.909175 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:52.909187 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:52.910618 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:52.912544 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:52.912553 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:52.912557 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:52.912779 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:52.913765 (+ 986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:52.913768 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:52.913770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:52.913859 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:52.914390 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:52.914399 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:52.914532 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 06:37:52.914653 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 06:37:52.914848 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":154,"threads_started":1}
I20260928 06:37:52.916232 18798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:37:52 dist-test-slave-j0qf krb5kdc[18565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
Sep 28 06:37:52 dist-test-slave-j0qf krb5kdc[18565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577472, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.60@KRBTEST.COM for kudu/127.5.22.61@KRBTEST.COM
I20260928 06:37:52.924468 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.914218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38176 (local address 127.5.22.62:43013)
0928 06:37:52.914589 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:52.914600 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:52.914609 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:37:52.914664 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:52.914666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:52.914712 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:52.914799 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:52.915683 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.916388 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.917404 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.917702 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.917786 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:52.920899 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:52.920903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:52.920905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:52.920931 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:52.922607 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.923062 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.923065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.923066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.923181 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.923491 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.923498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.923501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.923771 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:52.923999 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:52.924118 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 06:37:52.924255 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":154,"threads_started":1}
I20260928 06:37:52.926276 18802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.916858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:38605 (local address 127.0.0.1:39480)
0928 06:37:52.917058 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.917075 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:52.917153 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:52.917421 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:52.917425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:52.917636 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:52.917731 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:52.917736 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.918769 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.918772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:52.919706 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:52.919716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.920459 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.920463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:52.920554 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:52.921061 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:52.921072 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:52.922569 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:52.924479 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:52.924481 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:52.924483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:52.924750 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:52.925077 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:52.925080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:52.925081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:52.925202 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:52.925697 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:52.925700 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:52.925802 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 06:37:52.925922 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 06:37:52.926079 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260928 06:37:52.926426 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.916973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39480 (local address 127.5.22.61:38605)
0928 06:37:52.917137 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:52.917141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:52.917157 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:37:52.917236 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:52.917239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:52.917289 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:52.917389 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:52.917882 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.918643 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.919826 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.920236 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.920369 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:52.922720 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:52.922724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:52.922727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:52.922758 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:52.924364 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.924858 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.924862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.924864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.924952 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.925333 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.925336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.925338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.925582 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:52.925676 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:52.926108 (+ 432us) server_negotiation.cc:299] Negotiation successful
0928 06:37:52.926235 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 06:37:52.927644 18798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:37:52 dist-test-slave-j0qf krb5kdc[18565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.61@KRBTEST.COM
Sep 28 06:37:52 dist-test-slave-j0qf krb5kdc[18565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577472, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.60@KRBTEST.COM for kudu/127.5.22.60@KRBTEST.COM
I20260928 06:37:52.936110 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.925866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39482 (local address 127.5.22.61:38605)
0928 06:37:52.925997 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:52.926002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:52.926081 (+ 79us) server_negotiation.cc:407] Connection header received
0928 06:37:52.926181 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:52.926186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:52.926236 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:52.926340 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:52.927122 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.927817 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.928937 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.929415 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.929525 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:52.932607 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:52.932611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:52.932614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:52.932653 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:52.934452 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.934894 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.934898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.934900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.934981 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.935263 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.935268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.935269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.935469 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:52.935692 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:52.935835 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 06:37:52.935952 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 06:37:52.937042 18802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.928134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:32967 (local address 127.0.0.1:42388)
0928 06:37:52.928233 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:52.928246 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:52.928316 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:52.928538 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:52.928541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:52.928703 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:52.928790 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:52.928796 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.929773 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.929779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:52.930635 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:52.930648 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.931148 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.931152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:52.931227 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:52.931672 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:52.931683 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:52.933219 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:52.935425 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:52.935430 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:52.935432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:52.935626 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:52.935898 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:52.935901 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:52.935902 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:52.935982 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:52.936512 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:52.936514 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:52.936604 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 06:37:52.936709 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 06:37:52.936851 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 06:37:52.937052 18801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.928233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42388 (local address 127.5.22.60:32967)
0928 06:37:52.928347 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:52.928352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:52.928373 (+ 21us) server_negotiation.cc:407] Connection header received
0928 06:37:52.928400 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:52.928402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:52.928436 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:52.928498 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:52.928937 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.929665 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.930802 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.931025 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.931113 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:52.933356 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:52.933360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:52.933362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:52.933384 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:52.935314 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.935722 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.935728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.935730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.935804 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.936189 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.936191 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.936192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.936405 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:52.936511 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:52.936767 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 06:37:52.936919 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":49}
I20260928 06:37:52.940014 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:37:52.940738 18798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c35be34ef5345b69c150659f903a0fd: Bootstrap starting.
I20260928 06:37:52.943850 18798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c35be34ef5345b69c150659f903a0fd: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:52.944880 18798 log.cc:824] T 00000000000000000000000000000000 P 2c35be34ef5345b69c150659f903a0fd: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:52.947208 18798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c35be34ef5345b69c150659f903a0fd: No bootstrap required, opened a new log
I20260928 06:37:52.952072 18798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c35be34ef5345b69c150659f903a0fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43013 } } peers { permanent_uuid: "09291973153a4bd0ba2786dfb7c6ad7a" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 } } peers { permanent_uuid: "2c35be34ef5345b69c150659f903a0fd" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } }
I20260928 06:37:52.952402 18798 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c35be34ef5345b69c150659f903a0fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:52.952466 18798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c35be34ef5345b69c150659f903a0fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c35be34ef5345b69c150659f903a0fd, State: Initialized, Role: FOLLOWER
I20260928 06:37:52.952987 18798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c35be34ef5345b69c150659f903a0fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43013 } } peers { permanent_uuid: "09291973153a4bd0ba2786dfb7c6ad7a" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 } } peers { permanent_uuid: "2c35be34ef5345b69c150659f903a0fd" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } }
I20260928 06:37:52.953513 18801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.943052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42396 (local address 127.5.22.60:32967)
0928 06:37:52.943157 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:52.943161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:52.943724 (+ 563us) server_negotiation.cc:407] Connection header received
0928 06:37:52.943925 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:52.943931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:52.943980 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:52.944033 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:52.944990 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.945669 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.946806 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.947096 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.947167 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:52.949564 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:52.949568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:52.949570 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:52.949597 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:52.951404 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.951985 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.951990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.951992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.952088 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.952464 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.952471 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.952474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.952715 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:52.952825 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:52.953205 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 06:37:52.953349 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:37:52.953567 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.942652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39498 (local address 127.5.22.61:38605)
0928 06:37:52.942788 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:52.942792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:52.943058 (+ 266us) server_negotiation.cc:407] Connection header received
0928 06:37:52.943237 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:52.943240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:52.943282 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:52.943373 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:52.944479 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.945300 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.946684 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.947033 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.947112 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:52.949548 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:52.949551 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:52.949554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:52.949578 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:52.951530 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.952081 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.952088 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.952093 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.952183 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.952542 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.952545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.952547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.952765 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:52.952874 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:52.953296 (+ 422us) server_negotiation.cc:299] Negotiation successful
0928 06:37:52.953421 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:37:52.954376 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:52.942286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38188 (local address 127.5.22.62:43013)
0928 06:37:52.942442 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:52.942446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:52.943385 (+ 939us) server_negotiation.cc:407] Connection header received
0928 06:37:52.943550 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:52.943554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:52.943614 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:52.943699 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:52.944589 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.945365 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.946684 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:52.947045 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:52.947112 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:52.949536 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:52.949540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:52.949543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:52.949572 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:52.952422 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.952870 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.952874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.952877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.952981 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:52.953311 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:52.953318 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:52.953320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:52.953552 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:52.953673 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:52.954067 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 06:37:52.954187 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 06:37:52.955411 18798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c35be34ef5345b69c150659f903a0fd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:52.955439 18812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35be34ef5345b69c150659f903a0fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43013 } } peers { permanent_uuid: "09291973153a4bd0ba2786dfb7c6ad7a" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 } } peers { permanent_uuid: "2c35be34ef5345b69c150659f903a0fd" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } } }
I20260928 06:37:52.955595 18812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35be34ef5345b69c150659f903a0fd [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:37:52.961184 18823 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c35be34ef5345b69c150659f903a0fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:37:52.961273 18823 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:37:52.959249 18730 master_runner.cc:186] Error getting master registration for 127.5.22.62:43013: OK, instance_id { permanent_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" instance_seqno: 1790577472422497 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:37:52.964072 18730 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:37:53.035621 18722 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } attempt: 4
Sep 28 06:37:53 dist-test-slave-j0qf krb5kdc[18565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577472, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.61@KRBTEST.COM for kudu/127.5.22.60@KRBTEST.COM
I20260928 06:37:53.045257 18801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:53.036222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42398 (local address 127.5.22.60:32967)
0928 06:37:53.036367 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:53.036371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:53.036386 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:37:53.036525 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:53.036527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:53.036587 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:53.036661 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:53.037176 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:53.037803 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:53.038914 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:53.039149 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:53.039237 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:53.041624 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:53.041628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:53.041633 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:53.041675 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:53.043469 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:53.043944 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:53.043949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:53.043952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:53.044040 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:53.044350 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:53.044355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:53.044359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:53.044555 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:53.044676 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:53.044962 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 06:37:53.045086 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 06:37:53.045257 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:53.036159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:32967 (local address 127.0.0.1:42398)
0928 06:37:53.036326 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:53.036343 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:53.036428 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:53.036705 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:53.036708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:53.036932 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:53.037042 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:53.037048 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:53.037927 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:53.037931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:53.038792 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:53.038803 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:53.039259 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:53.039263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:53.039357 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:53.039883 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:53.039905 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:53.041456 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:53.043596 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:53.043603 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:53.043606 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:53.043832 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:53.044144 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:53.044149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:53.044151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:53.044250 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:53.044664 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:53.044668 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:53.044785 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 06:37:53.044903 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 06:37:53.045066 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 06:37:53.048213 18722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09291973153a4bd0ba2786dfb7c6ad7a: Bootstrap starting.
I20260928 06:37:53.051026 18722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09291973153a4bd0ba2786dfb7c6ad7a: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:53.051846 18722 log.cc:824] T 00000000000000000000000000000000 P 09291973153a4bd0ba2786dfb7c6ad7a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:53.053791 18722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09291973153a4bd0ba2786dfb7c6ad7a: No bootstrap required, opened a new log
I20260928 06:37:53.057277 18722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09291973153a4bd0ba2786dfb7c6ad7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43013 } } peers { permanent_uuid: "09291973153a4bd0ba2786dfb7c6ad7a" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 } } peers { permanent_uuid: "2c35be34ef5345b69c150659f903a0fd" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } }
I20260928 06:37:53.057570 18722 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09291973153a4bd0ba2786dfb7c6ad7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:53.057660 18722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09291973153a4bd0ba2786dfb7c6ad7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09291973153a4bd0ba2786dfb7c6ad7a, State: Initialized, Role: FOLLOWER
I20260928 06:37:53.058164 18722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09291973153a4bd0ba2786dfb7c6ad7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43013 } } peers { permanent_uuid: "09291973153a4bd0ba2786dfb7c6ad7a" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 } } peers { permanent_uuid: "2c35be34ef5345b69c150659f903a0fd" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } }
I20260928 06:37:53.058537 18646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } attempt: 4
I20260928 06:37:53.059509 18722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09291973153a4bd0ba2786dfb7c6ad7a [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 06:37:53 dist-test-slave-j0qf krb5kdc[18565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577472, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for kudu/127.5.22.60@KRBTEST.COM
W20260928 06:37:53.065559 18837 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09291973153a4bd0ba2786dfb7c6ad7a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:37:53.065706 18837 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:37:53.062613 18826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09291973153a4bd0ba2786dfb7c6ad7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43013 } } peers { permanent_uuid: "09291973153a4bd0ba2786dfb7c6ad7a" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 } } peers { permanent_uuid: "2c35be34ef5345b69c150659f903a0fd" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } } }
I20260928 06:37:53.065910 18826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09291973153a4bd0ba2786dfb7c6ad7a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:37:53.065227 18653 master_runner.cc:186] Error getting master registration for 127.5.22.62:43013: OK, instance_id { permanent_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" instance_seqno: 1790577472422497 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:37:53.069702 18653 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:37:53.070094 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:53.059007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:32967 (local address 127.0.0.1:42410)
0928 06:37:53.059201 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:53.059218 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:53.059301 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:53.059963 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:53.059966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:53.060178 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:53.060301 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:53.060311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:53.061489 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:53.061495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:53.062433 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:53.062448 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:53.063149 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:53.063153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:53.063250 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:53.063892 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:53.063909 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:53.065661 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:53.068199 (+ 2538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:53.068203 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:53.068205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:53.068496 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:53.068848 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:53.068852 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:53.068855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:53.068958 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:53.069499 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:53.069502 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:53.069618 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 06:37:53.069734 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 06:37:53.069903 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260928 06:37:53.070726 18801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:53.059551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42410 (local address 127.5.22.60:32967)
0928 06:37:53.059682 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:53.059686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:53.059703 (+ 17us) server_negotiation.cc:407] Connection header received
0928 06:37:53.059746 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:53.059749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:53.059793 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:53.059917 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:53.060441 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:53.061348 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:53.062640 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:53.063003 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:53.063106 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:53.065960 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:53.065967 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:53.065973 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:53.066005 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:53.068054 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:53.068626 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:53.068630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:53.068632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:53.068729 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:53.069077 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:53.069080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:53.069082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:53.069325 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:53.069445 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:53.070426 (+ 981us) server_negotiation.cc:299] Negotiation successful
0928 06:37:53.070558 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 06:37:53.073923 18646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4: Bootstrap starting.
I20260928 06:37:53.076821 18646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:53.077755 18646 log.cc:824] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:53.080258 18646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4: No bootstrap required, opened a new log
I20260928 06:37:53.085734 18646 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43013 } } peers { permanent_uuid: "09291973153a4bd0ba2786dfb7c6ad7a" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 } } peers { permanent_uuid: "2c35be34ef5345b69c150659f903a0fd" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } }
I20260928 06:37:53.085987 18646 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:53.086084 18646 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2ed6f6cfc4a4eefbc05ccc724fb76f4, State: Initialized, Role: FOLLOWER
I20260928 06:37:53.086603 18646 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43013 } } peers { permanent_uuid: "09291973153a4bd0ba2786dfb7c6ad7a" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 } } peers { permanent_uuid: "2c35be34ef5345b69c150659f903a0fd" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } }
I20260928 06:37:53.088068 18646 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:53.088429 18840 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43013 } } peers { permanent_uuid: "09291973153a4bd0ba2786dfb7c6ad7a" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 } } peers { permanent_uuid: "2c35be34ef5345b69c150659f903a0fd" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } } }
I20260928 06:37:53.088640 18840 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:37:53.093683 18851 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:37:53.093833 18851 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:37:53.096792 18578 master_runner.cc:263] No leader master found from master b2ed6f6cfc4a4eefbc05ccc724fb76f4
I20260928 06:37:53.096855 18578 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 06:37:53.132203 18674 tablet.cc:2369] T 00000000000000000000000000000000 P 09291973153a4bd0ba2786dfb7c6ad7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:37:53.137665 18751 tablet.cc:2369] T 00000000000000000000000000000000 P 2c35be34ef5345b69c150659f903a0fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:37:53.152199 18840 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:37:53.152441 18840 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43013 } } peers { permanent_uuid: "09291973153a4bd0ba2786dfb7c6ad7a" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 } } peers { permanent_uuid: "2c35be34ef5345b69c150659f903a0fd" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } }
I20260928 06:37:53.153635 18840 leader_election.cc:288] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09291973153a4bd0ba2786dfb7c6ad7a (127.5.22.61:38605), 2c35be34ef5345b69c150659f903a0fd (127.5.22.60:32967)
I20260928 06:37:53.154282 18772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c35be34ef5345b69c150659f903a0fd" is_pre_election: true
I20260928 06:37:53.154296 18695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09291973153a4bd0ba2786dfb7c6ad7a" is_pre_election: true
I20260928 06:37:53.154652 18772 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2c35be34ef5345b69c150659f903a0fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2ed6f6cfc4a4eefbc05ccc724fb76f4 in term 0.
I20260928 06:37:53.154670 18695 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 09291973153a4bd0ba2786dfb7c6ad7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2ed6f6cfc4a4eefbc05ccc724fb76f4 in term 0.
I20260928 06:37:53.155128 18597 leader_election.cc:302] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c35be34ef5345b69c150659f903a0fd, b2ed6f6cfc4a4eefbc05ccc724fb76f4; no voters:
I20260928 06:37:53.155550 18840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:37:53.155653 18840 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:37:53.155728 18840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:53.157160 18840 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43013 } } peers { permanent_uuid: "09291973153a4bd0ba2786dfb7c6ad7a" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 } } peers { permanent_uuid: "2c35be34ef5345b69c150659f903a0fd" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } }
I20260928 06:37:53.157541 18840 leader_election.cc:288] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [CANDIDATE]: Term 1 election: Requested vote from peers 09291973153a4bd0ba2786dfb7c6ad7a (127.5.22.61:38605), 2c35be34ef5345b69c150659f903a0fd (127.5.22.60:32967)
I20260928 06:37:53.157958 18695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09291973153a4bd0ba2786dfb7c6ad7a"
I20260928 06:37:53.158103 18695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09291973153a4bd0ba2786dfb7c6ad7a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:53.158298 18772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c35be34ef5345b69c150659f903a0fd"
I20260928 06:37:53.158476 18772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c35be34ef5345b69c150659f903a0fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:53.159530 18695 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 09291973153a4bd0ba2786dfb7c6ad7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2ed6f6cfc4a4eefbc05ccc724fb76f4 in term 1.
I20260928 06:37:53.159819 18772 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2c35be34ef5345b69c150659f903a0fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2ed6f6cfc4a4eefbc05ccc724fb76f4 in term 1.
I20260928 06:37:53.159858 18596 leader_election.cc:302] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09291973153a4bd0ba2786dfb7c6ad7a, b2ed6f6cfc4a4eefbc05ccc724fb76f4; no voters:
I20260928 06:37:53.161641 18840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:53.161921 18840 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [term 1 LEADER]: Becoming Leader. State: Replica: b2ed6f6cfc4a4eefbc05ccc724fb76f4, State: Running, Role: LEADER
I20260928 06:37:53.162494 18840 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43013 } } peers { permanent_uuid: "09291973153a4bd0ba2786dfb7c6ad7a" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 } } peers { permanent_uuid: "2c35be34ef5345b69c150659f903a0fd" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } }
I20260928 06:37:53.164896 18853 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2ed6f6cfc4a4eefbc05ccc724fb76f4. Latest consensus state: current_term: 1 leader_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43013 } } peers { permanent_uuid: "09291973153a4bd0ba2786dfb7c6ad7a" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 } } peers { permanent_uuid: "2c35be34ef5345b69c150659f903a0fd" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } } }
I20260928 06:37:53.165097 18853 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:53.165545 18855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:53.166676 18855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260928 06:37:53.169108 18599 tablet.cc:2369] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:37:53.172035 18772 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c35be34ef5345b69c150659f903a0fd [term 1 FOLLOWER]: Refusing update from remote peer b2ed6f6cfc4a4eefbc05ccc724fb76f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:53.172058 18695 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 09291973153a4bd0ba2786dfb7c6ad7a [term 1 FOLLOWER]: Refusing update from remote peer b2ed6f6cfc4a4eefbc05ccc724fb76f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:53.172776 18853 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09291973153a4bd0ba2786dfb7c6ad7a" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:53.173000 18840 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c35be34ef5345b69c150659f903a0fd" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:53.176748 18826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09291973153a4bd0ba2786dfb7c6ad7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2ed6f6cfc4a4eefbc05ccc724fb76f4. Latest consensus state: current_term: 1 leader_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43013 } } peers { permanent_uuid: "09291973153a4bd0ba2786dfb7c6ad7a" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 } } peers { permanent_uuid: "2c35be34ef5345b69c150659f903a0fd" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } } }
I20260928 06:37:53.176899 18826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09291973153a4bd0ba2786dfb7c6ad7a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:37:53.177037 18812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35be34ef5345b69c150659f903a0fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2ed6f6cfc4a4eefbc05ccc724fb76f4. Latest consensus state: current_term: 1 leader_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43013 } } peers { permanent_uuid: "09291973153a4bd0ba2786dfb7c6ad7a" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 } } peers { permanent_uuid: "2c35be34ef5345b69c150659f903a0fd" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } } }
I20260928 06:37:53.177147 18812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35be34ef5345b69c150659f903a0fd [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:37:53.178809 18840 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43013 } } peers { permanent_uuid: "09291973153a4bd0ba2786dfb7c6ad7a" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 } } peers { permanent_uuid: "2c35be34ef5345b69c150659f903a0fd" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } } }
I20260928 06:37:53.178930 18840 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:53.179050 18856 mvcc.cc:205] Tried to move back new op lower bound from 7334205330105946112 to 7334205330077020160. Current Snapshot: MvccSnapshot[applied={T|T < 7334205330105946112}]
I20260928 06:37:53.179886 18826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09291973153a4bd0ba2786dfb7c6ad7a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43013 } } peers { permanent_uuid: "09291973153a4bd0ba2786dfb7c6ad7a" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 } } peers { permanent_uuid: "2c35be34ef5345b69c150659f903a0fd" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } } }
I20260928 06:37:53.180015 18826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09291973153a4bd0ba2786dfb7c6ad7a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:37:53.181100 18853 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43013 } } peers { permanent_uuid: "09291973153a4bd0ba2786dfb7c6ad7a" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 } } peers { permanent_uuid: "2c35be34ef5345b69c150659f903a0fd" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } } }
I20260928 06:37:53.181216 18853 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:53.182273 18855 catalog_manager.cc:1347] Generated new cluster ID: 3d65413796e845bcac373922f92048d4
I20260928 06:37:53.182363 18855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:53.182390 18812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35be34ef5345b69c150659f903a0fd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed6f6cfc4a4eefbc05ccc724fb76f4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43013 } } peers { permanent_uuid: "09291973153a4bd0ba2786dfb7c6ad7a" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 38605 } } peers { permanent_uuid: "2c35be34ef5345b69c150659f903a0fd" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 32967 } } }
I20260928 06:37:53.182507 18812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35be34ef5345b69c150659f903a0fd [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:37:53.186822 18858 mvcc.cc:205] Tried to move back new op lower bound from 7334205330105946112 to 7334205330077020160. Current Snapshot: MvccSnapshot[applied={T|T < 7334205330105946112}]
I20260928 06:37:53.188869 18857 mvcc.cc:205] Tried to move back new op lower bound from 7334205330105946112 to 7334205330077020160. Current Snapshot: MvccSnapshot[applied={T|T < 7334205330105946112}]
I20260928 06:37:53.205327 18855 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:53.206701 18855 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:53.216859 18855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2ed6f6cfc4a4eefbc05ccc724fb76f4: Generated new TSK 0
I20260928 06:37:53.218555 18855 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:37:53.962642 18823 catalog_manager.cc:1259] Loaded cluster ID: 3d65413796e845bcac373922f92048d4
I20260928 06:37:53.962754 18823 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2c35be34ef5345b69c150659f903a0fd: loading cluster ID for follower catalog manager: success
I20260928 06:37:53.964506 18823 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2c35be34ef5345b69c150659f903a0fd: acquiring CA information for follower catalog manager: success
I20260928 06:37:53.965652 18823 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2c35be34ef5345b69c150659f903a0fd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:37:54.067122 18837 catalog_manager.cc:1259] Loaded cluster ID: 3d65413796e845bcac373922f92048d4
I20260928 06:37:54.067215 18837 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 09291973153a4bd0ba2786dfb7c6ad7a: loading cluster ID for follower catalog manager: success
I20260928 06:37:54.068851 18837 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 09291973153a4bd0ba2786dfb7c6ad7a: acquiring CA information for follower catalog manager: success
I20260928 06:37:54.070020 18837 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 09291973153a4bd0ba2786dfb7c6ad7a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:37:54.291544 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 18578
I20260928 06:37:54.298280 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 18653
I20260928 06:37:54.304813 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 18730
2026-09-28T06:37:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3245 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:54 dist-test-slave-j0qf krb5kdc[18870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:54 dist-test-slave-j0qf krb5kdc[18870](info): set up 2 sockets
Sep 28 06:37:54 dist-test-slave-j0qf krb5kdc[18870](info): commencing operation
krb5kdc: starting...
W20260928 06:37:55.380940 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:37:55 dist-test-slave-j0qf krb5kdc[18870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:37:55.525455 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:34281
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:33233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:34281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:55.636896 18883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:55.637187 18883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:55.637228 18883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:55.641265 18883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:55.641327 18883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:55.641347 18883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:55.641363 18883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:55.641379 18883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:55.646679 18883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:33233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:34281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:34281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:55.647692 18883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:55.648663 18883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:55.657474 18892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:55.657634 18883 server_base.cc:1034] running on GCE node
W20260928 06:37:55.657446 18889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:55.657461 18890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:55.658384 18883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:55.659549 18883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:55.660737 18883 hybrid_clock.cc:648] HybridClock initialized: now 1790577475660723 us; error 26 us; skew 500 ppm
Sep 28 06:37:55 dist-test-slave-j0qf krb5kdc[18870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577475, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:55.663638 18883 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:37:55.664561 18883 webserver.cc:460] Webserver started at http://127.5.22.62:37967/ using document root <none> and password file <none>
I20260928 06:37:55.665094 18883 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:55.665153 18883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:55.665318 18883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:55.667476 18883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "189fabaa8db9415c91a4195f4ffd47e1"
format_stamp: "Formatted at 2026-09-28 06:37:55 on dist-test-slave-j0qf"
server_key: "8817b2ff7ea630a38a7bf34398da3a59"
server_key_iv: "058b426ad5d2a47fbbe9abffc33a4a66"
server_key_version: "encryptionkey@0"
I20260928 06:37:55.668018 18883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "189fabaa8db9415c91a4195f4ffd47e1"
format_stamp: "Formatted at 2026-09-28 06:37:55 on dist-test-slave-j0qf"
server_key: "8817b2ff7ea630a38a7bf34398da3a59"
server_key_iv: "058b426ad5d2a47fbbe9abffc33a4a66"
server_key_version: "encryptionkey@0"
I20260928 06:37:55.671830 18883 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 06:37:55.674265 18899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:55.675534 18883 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:37:55.675647 18883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "189fabaa8db9415c91a4195f4ffd47e1"
format_stamp: "Formatted at 2026-09-28 06:37:55 on dist-test-slave-j0qf"
server_key: "8817b2ff7ea630a38a7bf34398da3a59"
server_key_iv: "058b426ad5d2a47fbbe9abffc33a4a66"
server_key_version: "encryptionkey@0"
I20260928 06:37:55.675770 18883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:55.696753 18883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:55.700090 18883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:55.700242 18883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:55.707079 18883 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:34281
I20260928 06:37:55.707105 18952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:34281 every 8 connection(s)
I20260928 06:37:55.708407 18883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:55.711160 18953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:55.711840 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 18883
I20260928 06:37:55.711989 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:37:55.712287 5208 external_mini_cluster.cc:1428] Setting key a23d98d5548c1a89a051d969b2f01073
I20260928 06:37:55.717905 18953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 189fabaa8db9415c91a4195f4ffd47e1: Bootstrap starting.
I20260928 06:37:55.720574 18953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 189fabaa8db9415c91a4195f4ffd47e1: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:55.721362 18953 log.cc:824] T 00000000000000000000000000000000 P 189fabaa8db9415c91a4195f4ffd47e1: Log is configured to *not* fsync() on all Append() calls
Sep 28 06:37:55 dist-test-slave-j0qf krb5kdc[18870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:37:55.723625 18953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 189fabaa8db9415c91a4195f4ffd47e1: No bootstrap required, opened a new log
I20260928 06:37:55.727069 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:55.713651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50214 (local address 127.5.22.62:34281)
0928 06:37:55.714078 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:55.714097 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:55.714145 (+ 48us) server_negotiation.cc:407] Connection header received
0928 06:37:55.714986 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:55.715014 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:55.715434 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:55.715818 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:55.716585 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:55.718047 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:55.718926 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:55.719625 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:55.719792 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:55.722482 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:55.722493 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:55.722507 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:55.722535 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:55.724228 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:55.724687 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:55.724692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:55.724696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:55.724806 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:55.725098 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:55.725102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:55.725103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:55.725516 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:55.725718 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:55.726004 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 06:37:55.726286 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":112,"threads_started":1}
I20260928 06:37:55.727178 18953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 189fabaa8db9415c91a4195f4ffd47e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189fabaa8db9415c91a4195f4ffd47e1" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34281 } }
I20260928 06:37:55.727458 18953 raft_consensus.cc:374] T 00000000000000000000000000000000 P 189fabaa8db9415c91a4195f4ffd47e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:55.727545 18953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 189fabaa8db9415c91a4195f4ffd47e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 189fabaa8db9415c91a4195f4ffd47e1, State: Initialized, Role: FOLLOWER
I20260928 06:37:55.728030 18953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 189fabaa8db9415c91a4195f4ffd47e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189fabaa8db9415c91a4195f4ffd47e1" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34281 } }
I20260928 06:37:55.728178 18953 raft_consensus.cc:388] T 00000000000000000000000000000000 P 189fabaa8db9415c91a4195f4ffd47e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:55.728248 18953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 189fabaa8db9415c91a4195f4ffd47e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:55.728344 18953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 189fabaa8db9415c91a4195f4ffd47e1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:55.729322 18953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 189fabaa8db9415c91a4195f4ffd47e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189fabaa8db9415c91a4195f4ffd47e1" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34281 } }
I20260928 06:37:55.729663 18953 leader_election.cc:302] T 00000000000000000000000000000000 P 189fabaa8db9415c91a4195f4ffd47e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 189fabaa8db9415c91a4195f4ffd47e1; no voters:
I20260928 06:37:55.729998 18953 leader_election.cc:288] T 00000000000000000000000000000000 P 189fabaa8db9415c91a4195f4ffd47e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:55.730063 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:37:55.730150 18958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 189fabaa8db9415c91a4195f4ffd47e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:55.730345 18958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 189fabaa8db9415c91a4195f4ffd47e1 [term 1 LEADER]: Becoming Leader. State: Replica: 189fabaa8db9415c91a4195f4ffd47e1, State: Running, Role: LEADER
I20260928 06:37:55.730650 18958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 189fabaa8db9415c91a4195f4ffd47e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189fabaa8db9415c91a4195f4ffd47e1" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34281 } }
I20260928 06:37:55.731237 18953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 189fabaa8db9415c91a4195f4ffd47e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:55.731838 18958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 189fabaa8db9415c91a4195f4ffd47e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 189fabaa8db9415c91a4195f4ffd47e1. Latest consensus state: current_term: 1 leader_uuid: "189fabaa8db9415c91a4195f4ffd47e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189fabaa8db9415c91a4195f4ffd47e1" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34281 } } }
I20260928 06:37:55.731979 18958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 189fabaa8db9415c91a4195f4ffd47e1 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:55.732308 18959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 189fabaa8db9415c91a4195f4ffd47e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "189fabaa8db9415c91a4195f4ffd47e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189fabaa8db9415c91a4195f4ffd47e1" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34281 } } }
I20260928 06:37:55.732471 18965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:55.732481 18959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 189fabaa8db9415c91a4195f4ffd47e1 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:55.735714 18965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:55.741580 18965 catalog_manager.cc:1347] Generated new cluster ID: 177be0919ee04c5e9c6d8053a47c2d59
I20260928 06:37:55.741669 18965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:55.747874 18965 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:55.748800 18965 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:55.764863 18965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 189fabaa8db9415c91a4195f4ffd47e1: Generated new TSK 0
I20260928 06:37:55.765600 18965 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:37:55.786837 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 18883
2026-09-28T06:37:55Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1480 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:55 dist-test-slave-j0qf krb5kdc[18980](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:55 dist-test-slave-j0qf krb5kdc[18980](info): set up 2 sockets
Sep 28 06:37:55 dist-test-slave-j0qf krb5kdc[18980](info): commencing operation
krb5kdc: starting...
W20260928 06:37:56.813117 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:37:56 dist-test-slave-j0qf krb5kdc[18980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:37:56.941924 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:39465
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:41197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:39465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:57.054910 18994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:57.055229 18994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:57.055313 18994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:57.059307 18994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:57.059392 18994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:57.059427 18994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:57.059470 18994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:57.059504 18994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:57.064807 18994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:41197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:39465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:39465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:57.065956 18994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:57.067008 18994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:57.075583 18999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:57.075620 19002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:57.075623 19000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:57.075798 18994 server_base.cc:1034] running on GCE node
I20260928 06:37:57.076401 18994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:57.077533 18994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:57.078715 18994 hybrid_clock.cc:648] HybridClock initialized: now 1790577477078688 us; error 43 us; skew 500 ppm
Sep 28 06:37:57 dist-test-slave-j0qf krb5kdc[18980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577477, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:57.085670 18994 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:37:57.086709 18994 webserver.cc:460] Webserver started at http://127.5.22.62:45167/ using document root <none> and password file <none>
I20260928 06:37:57.087304 18994 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:57.087399 18994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:57.087633 18994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:57.089717 18994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "34e41488fd0749dd8517addf86daae63"
format_stamp: "Formatted at 2026-09-28 06:37:57 on dist-test-slave-j0qf"
server_key: "8f751842537442d359b56869617f1c54"
server_key_iv: "f6219250bb9897a38249920182d79756"
server_key_version: "encryptionkey@0"
I20260928 06:37:57.090307 18994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "34e41488fd0749dd8517addf86daae63"
format_stamp: "Formatted at 2026-09-28 06:37:57 on dist-test-slave-j0qf"
server_key: "8f751842537442d359b56869617f1c54"
server_key_iv: "f6219250bb9897a38249920182d79756"
server_key_version: "encryptionkey@0"
I20260928 06:37:57.094089 18994 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:37:57.096621 19009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:57.097900 18994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:37:57.098073 18994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "34e41488fd0749dd8517addf86daae63"
format_stamp: "Formatted at 2026-09-28 06:37:57 on dist-test-slave-j0qf"
server_key: "8f751842537442d359b56869617f1c54"
server_key_iv: "f6219250bb9897a38249920182d79756"
server_key_version: "encryptionkey@0"
I20260928 06:37:57.098186 18994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:57.118871 18994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:57.122001 18994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:57.122221 18994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:57.129208 18994 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:39465
I20260928 06:37:57.129498 19061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:39465 every 8 connection(s)
I20260928 06:37:57.130195 18994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:57.133052 19062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:57.138448 19062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63: Bootstrap starting.
I20260928 06:37:57.138875 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 18994
I20260928 06:37:57.139006 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:37:57.139328 5208 external_mini_cluster.cc:1428] Setting key a55f3268795e68f9739f42434b55367e
I20260928 06:37:57.141448 19062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:57.142199 19062 log.cc:824] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:57.144273 19062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63: No bootstrap required, opened a new log
Sep 28 06:37:57 dist-test-slave-j0qf krb5kdc[18980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:37:57.147905 19062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e41488fd0749dd8517addf86daae63" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 39465 } }
I20260928 06:37:57.148164 19062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:57.148258 19062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e41488fd0749dd8517addf86daae63, State: Initialized, Role: FOLLOWER
I20260928 06:37:57.148770 19062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e41488fd0749dd8517addf86daae63" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 39465 } }
I20260928 06:37:57.148959 19062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:57.149034 19062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:57.149134 19062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:57.150390 19062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e41488fd0749dd8517addf86daae63" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 39465 } }
I20260928 06:37:57.150779 19062 leader_election.cc:302] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34e41488fd0749dd8517addf86daae63; no voters:
I20260928 06:37:57.151129 19062 leader_election.cc:288] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:57.151248 19068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:57.151495 19068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [term 1 LEADER]: Becoming Leader. State: Replica: 34e41488fd0749dd8517addf86daae63, State: Running, Role: LEADER
I20260928 06:37:57.151861 19068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e41488fd0749dd8517addf86daae63" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 39465 } }
I20260928 06:37:57.152541 19062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:57.153200 19070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34e41488fd0749dd8517addf86daae63. Latest consensus state: current_term: 1 leader_uuid: "34e41488fd0749dd8517addf86daae63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e41488fd0749dd8517addf86daae63" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 39465 } } }
I20260928 06:37:57.153332 19070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:57.154026 19069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34e41488fd0749dd8517addf86daae63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e41488fd0749dd8517addf86daae63" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 39465 } } }
I20260928 06:37:57.154127 19069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:57.154091 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:57.140680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.5.22.62:39465)
0928 06:37:57.141104 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:57.141120 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:57.141155 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:37:57.141836 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:57.141854 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:57.142178 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:57.142526 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:57.143316 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.144311 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.145195 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.145666 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.145847 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:57.148466 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:57.148472 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:57.148489 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:57.148522 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:57.150700 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:57.151257 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:57.151263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:57.151268 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:57.151419 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:57.151753 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:57.151757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:57.151758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:57.152263 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:57.152463 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:57.152796 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 06:37:57.153155 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":157,"threads_started":1}
W20260928 06:37:57.157631 19083 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:37:57.157711 19083 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:37:57.157819 19074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:57.158744 19074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:57.164042 19074 catalog_manager.cc:1347] Generated new cluster ID: 6c0d22fc78814b4da244d4b25fd051f8
I20260928 06:37:57.164111 19074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:57.179315 19074 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:57.180277 19074 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:57.184770 19074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63: Generated new TSK 0
I20260928 06:37:57.185539 19074 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.22.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
I20260928 06:37:57.222787 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:39465
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:41197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:57.334453 19089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:57.334721 19089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:57.334767 19089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:57.338766 19089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:57.338817 19089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:57.338919 19089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:37:57.344048 19089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:41197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:39465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:57.345096 19089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:57.346127 19089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:57.356130 19094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:57.356139 19097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:57.356139 19095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:57.356426 19089 server_base.cc:1034] running on GCE node
I20260928 06:37:57.356839 19089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:57.357513 19089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:57.358697 19089 hybrid_clock.cc:648] HybridClock initialized: now 1790577477358690 us; error 30 us; skew 500 ppm
Sep 28 06:37:57 dist-test-slave-j0qf krb5kdc[18980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577477, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:57.361356 19089 init.cc:377] Logged in from keytab as kudu/127.5.22.1@KRBTEST.COM (short username kudu)
I20260928 06:37:57.362269 19089 webserver.cc:460] Webserver started at http://127.5.22.1:45035/ using document root <none> and password file <none>
I20260928 06:37:57.362844 19089 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:57.362942 19089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:57.363171 19089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:57.365175 19089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "8afe0ccc5aaf41e68c49622dc18d96ae"
format_stamp: "Formatted at 2026-09-28 06:37:57 on dist-test-slave-j0qf"
server_key: "6189a654d4afaa2a89fc2da02c880f2c"
server_key_iv: "393a0a9c61976d80018652eb2fa3e9ad"
server_key_version: "encryptionkey@0"
I20260928 06:37:57.365723 19089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "8afe0ccc5aaf41e68c49622dc18d96ae"
format_stamp: "Formatted at 2026-09-28 06:37:57 on dist-test-slave-j0qf"
server_key: "6189a654d4afaa2a89fc2da02c880f2c"
server_key_iv: "393a0a9c61976d80018652eb2fa3e9ad"
server_key_version: "encryptionkey@0"
I20260928 06:37:57.369333 19089 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 06:37:57.371733 19104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:57.372982 19089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:37:57.373123 19089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "8afe0ccc5aaf41e68c49622dc18d96ae"
format_stamp: "Formatted at 2026-09-28 06:37:57 on dist-test-slave-j0qf"
server_key: "6189a654d4afaa2a89fc2da02c880f2c"
server_key_iv: "393a0a9c61976d80018652eb2fa3e9ad"
server_key_version: "encryptionkey@0"
I20260928 06:37:57.373234 19089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:57.392545 19089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:57.395565 19089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:57.395776 19089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:57.396374 19089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:57.397217 19089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:57.397287 19089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:57.397360 19089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:57.397420 19089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:57.406838 19089 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:40803
I20260928 06:37:57.406891 19218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:40803 every 8 connection(s)
I20260928 06:37:57.407765 19089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:37:57.409827 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 19089
I20260928 06:37:57.409977 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:37:57.410259 5208 external_mini_cluster.cc:1428] Setting key 4ba38c7efe858000a3d6078a06a22506
Sep 28 06:37:57 dist-test-slave-j0qf krb5kdc[18980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577477, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.22.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.2@KRBTEST.COM" created.
I20260928 06:37:57.424202 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:57.409608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:58067 (local address 127.5.22.62:39465)
0928 06:37:57.409762 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:57.409766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:57.410734 (+ 968us) server_negotiation.cc:407] Connection header received
0928 06:37:57.412214 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:57.412218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:57.412315 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:57.412450 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:57.414994 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.415707 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.417003 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.417249 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.417356 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:57.420292 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:57.420295 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:57.420297 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:57.420321 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:57.421714 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:57.422201 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:57.422203 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:57.422205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:57.422276 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:57.422758 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:57.422760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:57.422761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:57.422937 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:57.423010 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:57.423775 (+ 765us) server_negotiation.cc:299] Negotiation successful
0928 06:37:57.423937 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 06:37:57.424832 19221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:57.410092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.1:58067)
0928 06:37:57.410570 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:57.410624 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:57.411962 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:57.413495 (+ 1533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:57.413504 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:57.414060 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:57.414769 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:57.414793 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.415842 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.415846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:57.416862 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:57.416872 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.417412 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.417417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:57.417589 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:57.418218 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:57.418235 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:57.420112 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:57.421829 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:57.421833 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:57.421846 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:57.422104 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:57.422467 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:57.422469 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:57.422471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:57.422662 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:57.423058 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:57.423083 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:57.423472 (+ 389us) client_negotiation.cc:769] Sending connection context
0928 06:37:57.423677 (+ 205us) client_negotiation.cc:240] Negotiation successful
0928 06:37:57.423946 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":165,"threads_started":1}
I20260928 06:37:57.426280 19219 heartbeater.cc:347] Connected to a master server at 127.5.22.62:39465
I20260928 06:37:57.426594 19219 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:57.427199 19219 heartbeater.cc:511] Master 127.5.22.62:39465 requested a full tablet report, sending...
I20260928 06:37:57.428826 19026 ts_manager.cc:194] Registered new tserver with Master: 8afe0ccc5aaf41e68c49622dc18d96ae (127.5.22.1:40803)
I20260928 06:37:57.429940 19026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.1@KRBTEST.COM'} at 127.5.22.1:58067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
I20260928 06:37:57.441686 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:39465
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:41197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:57.553296 19224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:57.553622 19224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:57.553665 19224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:57.557755 19224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:57.557806 19224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:57.557863 19224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:37:57.562924 19224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:41197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:39465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:57.563861 19224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:57.564805 19224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:57.574260 19229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:57.574260 19232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:57.574397 19224 server_base.cc:1034] running on GCE node
W20260928 06:37:57.574404 19230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:57.574961 19224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:57.575596 19224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:57.576763 19224 hybrid_clock.cc:648] HybridClock initialized: now 1790577477576736 us; error 40 us; skew 500 ppm
Sep 28 06:37:57 dist-test-slave-j0qf krb5kdc[18980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577477, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:57.579674 19224 init.cc:377] Logged in from keytab as kudu/127.5.22.2@KRBTEST.COM (short username kudu)
I20260928 06:37:57.580580 19224 webserver.cc:460] Webserver started at http://127.5.22.2:40281/ using document root <none> and password file <none>
I20260928 06:37:57.581143 19224 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:57.581235 19224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:57.581470 19224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:57.583562 19224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513"
format_stamp: "Formatted at 2026-09-28 06:37:57 on dist-test-slave-j0qf"
server_key: "dd1a1d571a7462c99551d426c968baac"
server_key_iv: "08f6c7a64c0290f9de31ce4774d8f82c"
server_key_version: "encryptionkey@0"
I20260928 06:37:57.584153 19224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513"
format_stamp: "Formatted at 2026-09-28 06:37:57 on dist-test-slave-j0qf"
server_key: "dd1a1d571a7462c99551d426c968baac"
server_key_iv: "08f6c7a64c0290f9de31ce4774d8f82c"
server_key_version: "encryptionkey@0"
I20260928 06:37:57.588060 19224 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 06:37:57.590485 19239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:57.591717 19224 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 06:37:57.591856 19224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513"
format_stamp: "Formatted at 2026-09-28 06:37:57 on dist-test-slave-j0qf"
server_key: "dd1a1d571a7462c99551d426c968baac"
server_key_iv: "08f6c7a64c0290f9de31ce4774d8f82c"
server_key_version: "encryptionkey@0"
I20260928 06:37:57.591996 19224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:57.606017 19224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:57.608980 19224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:57.609186 19224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:57.609769 19224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:57.610631 19224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:57.610708 19224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:57.610780 19224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:57.610814 19224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:57.619527 19224 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:32797
I20260928 06:37:57.619551 19353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:32797 every 8 connection(s)
I20260928 06:37:57.620502 19224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:37:57.628252 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 19224
I20260928 06:37:57.628417 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:37:57.628707 5208 external_mini_cluster.cc:1428] Setting key f730377d305e48e3bf7bfe0ce3429086
Sep 28 06:37:57 dist-test-slave-j0qf krb5kdc[18980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577477, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:37:57.635116 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:57.622358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:53767 (local address 127.5.22.62:39465)
0928 06:37:57.622520 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:57.622524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:57.623343 (+ 819us) server_negotiation.cc:407] Connection header received
0928 06:37:57.624680 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:57.624684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:57.624740 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:57.624846 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:57.626037 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.626648 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.627801 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.628113 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.628246 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:57.631125 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:57.631132 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:57.631139 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:57.631172 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:57.632718 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:57.633354 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:57.633357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:57.633358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:57.633432 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:57.633816 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:57.633818 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:57.633819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:57.634075 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:57.634184 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:57.634873 (+ 689us) server_negotiation.cc:299] Negotiation successful
0928 06:37:57.634994 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 06:37:57.636436 19356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:57.622748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.2:53767)
0928 06:37:57.623192 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:57.623236 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:57.624465 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:57.625023 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:57.625030 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:57.625382 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:57.625890 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:57.625905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.626776 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.626779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:57.627672 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:57.627681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.628298 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.628301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:57.628555 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:57.629265 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:57.629286 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:57.630958 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:57.632841 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:57.632849 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:57.632867 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:57.633254 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:57.633524 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:57.633530 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:57.633535 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:57.633723 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:57.634198 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:57.634207 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:57.634580 (+ 373us) client_negotiation.cc:769] Sending connection context
0928 06:37:57.634809 (+ 229us) client_negotiation.cc:240] Negotiation successful
0928 06:37:57.635190 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":122,"threads_started":1}
I20260928 06:37:57.637760 19354 heartbeater.cc:347] Connected to a master server at 127.5.22.62:39465
I20260928 06:37:57.638023 19354 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:57.638571 19354 heartbeater.cc:511] Master 127.5.22.62:39465 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.22.3@KRBTEST.COM; defaulting to no policy
I20260928 06:37:57.639648 19026 ts_manager.cc:194] Registered new tserver with Master: ba8bcffaeb5a4a4eafe60c4a77c6f513 (127.5.22.2:32797)
I20260928 06:37:57.640259 19026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.2@KRBTEST.COM'} at 127.5.22.2:53767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
I20260928 06:37:57.656672 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:39465
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:41197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:57.767531 19359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:57.767809 19359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:57.767853 19359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:57.772554 19359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:57.772617 19359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:57.772677 19359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:37:57.778115 19359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:41197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:39465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:57.779230 19359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:57.780215 19359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:57.789956 19364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:57.790113 19367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:57.789956 19365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:57.790236 19359 server_base.cc:1034] running on GCE node
I20260928 06:37:57.790813 19359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:57.791460 19359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:57.792665 19359 hybrid_clock.cc:648] HybridClock initialized: now 1790577477792651 us; error 30 us; skew 500 ppm
Sep 28 06:37:57 dist-test-slave-j0qf krb5kdc[18980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577477, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:57.795533 19359 init.cc:377] Logged in from keytab as kudu/127.5.22.3@KRBTEST.COM (short username kudu)
I20260928 06:37:57.796559 19359 webserver.cc:460] Webserver started at http://127.5.22.3:42499/ using document root <none> and password file <none>
I20260928 06:37:57.797118 19359 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:57.797176 19359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:57.797405 19359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:57.799472 19359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "0bc647d664234b94808b88315a39b22f"
format_stamp: "Formatted at 2026-09-28 06:37:57 on dist-test-slave-j0qf"
server_key: "40911fae3b63f3fe72cd46c8e1047756"
server_key_iv: "e73d379fcd449c1e54699268b6697522"
server_key_version: "encryptionkey@0"
I20260928 06:37:57.800045 19359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "0bc647d664234b94808b88315a39b22f"
format_stamp: "Formatted at 2026-09-28 06:37:57 on dist-test-slave-j0qf"
server_key: "40911fae3b63f3fe72cd46c8e1047756"
server_key_iv: "e73d379fcd449c1e54699268b6697522"
server_key_version: "encryptionkey@0"
I20260928 06:37:57.803916 19359 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:37:57.806296 19374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:57.807561 19359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:37:57.807698 19359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "0bc647d664234b94808b88315a39b22f"
format_stamp: "Formatted at 2026-09-28 06:37:57 on dist-test-slave-j0qf"
server_key: "40911fae3b63f3fe72cd46c8e1047756"
server_key_iv: "e73d379fcd449c1e54699268b6697522"
server_key_version: "encryptionkey@0"
I20260928 06:37:57.807809 19359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:57.831955 19359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:57.835031 19359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:57.835235 19359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:57.835847 19359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:57.836675 19359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:57.836721 19359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:57.836787 19359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:57.836822 19359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:57.845974 19359 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:34975
I20260928 06:37:57.846000 19488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:34975 every 8 connection(s)
I20260928 06:37:57.846884 19359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:37:57.853139 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 19359
I20260928 06:37:57.853296 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
I20260928 06:37:57.853608 5208 external_mini_cluster.cc:1428] Setting key 6abb35841149d9d458e76ce2cb2e5d7c
Sep 28 06:37:57 dist-test-slave-j0qf krb5kdc[18980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577477, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:37:57.861894 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:57.848852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:37371 (local address 127.5.22.62:39465)
0928 06:37:57.849030 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:57.849035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:57.849871 (+ 836us) server_negotiation.cc:407] Connection header received
0928 06:37:57.851180 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:57.851182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:57.851230 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:57.851305 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:57.852514 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.853185 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.854912 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.855190 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.855270 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:57.858225 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:57.858230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:57.858232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:57.858258 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:57.859850 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:57.860387 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:57.860389 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:57.860390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:57.860504 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:57.860822 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:57.860825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:57.860826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:57.861011 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:57.861093 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:57.861636 (+ 543us) server_negotiation.cc:299] Negotiation successful
0928 06:37:57.861743 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 06:37:57.862803 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:57.849271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.3:37371)
0928 06:37:57.849713 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:57.849755 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:57.850980 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:57.851477 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:57.851484 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:57.851829 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:57.852380 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:57.852400 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.853382 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.853385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:57.854793 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:57.854805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.855344 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.855348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:57.855553 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:57.856161 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:57.856178 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:57.858035 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:57.859982 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:57.859987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:57.859999 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:57.860284 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:57.860592 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:57.860594 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:57.860595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:57.860733 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:57.861124 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:57.861129 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:57.861368 (+ 239us) client_negotiation.cc:769] Sending connection context
0928 06:37:57.861535 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 06:37:57.861792 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":122,"threads_started":1}
I20260928 06:37:57.864001 19489 heartbeater.cc:347] Connected to a master server at 127.5.22.62:39465
I20260928 06:37:57.864205 19489 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:57.864673 19489 heartbeater.cc:511] Master 127.5.22.62:39465 requested a full tablet report, sending...
I20260928 06:37:57.865702 19026 ts_manager.cc:194] Registered new tserver with Master: 0bc647d664234b94808b88315a39b22f (127.5.22.3:34975)
I20260928 06:37:57.866400 19026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.3@KRBTEST.COM'} at 127.5.22.3:37371
I20260928 06:37:57.867596 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:37:57.876938 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:57.869628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55592 (local address 127.5.22.62:39465)
0928 06:37:57.869741 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:57.869744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:57.869976 (+ 232us) server_negotiation.cc:407] Connection header received
0928 06:37:57.870073 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:57.870075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:57.870111 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:57.870176 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:57.870899 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.871485 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.872327 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.872540 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.872615 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:57.873902 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:57.873906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:57.873908 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:57.873929 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:57.875344 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:57.875741 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:57.875744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:57.875745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:57.875821 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:57.876115 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:57.876117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:57.876118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:57.876318 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:57.876419 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:57.876634 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 06:37:57.876724 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:37:57.879819 19026 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:37:57.882848 19026 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:37:57.897892 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:57.891801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:40803 (local address 127.0.0.1:41034)
0928 06:37:57.892148 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:57.892188 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:57.892336 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:57.893536 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:57.893545 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:57.893576 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:57.893716 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:57.893724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.895111 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.895114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:57.896403 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:57.896412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.897345 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.897348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:57.897462 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:57.897497 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 06:37:57.897616 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 06:37:57.897738 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":32,"thread_start_us":87,"threads_started":1}
I20260928 06:37:57.897907 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:57.892549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41034 (local address 127.5.22.1:40803)
0928 06:37:57.892960 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:57.892976 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:57.893006 (+ 30us) server_negotiation.cc:407] Connection header received
0928 06:37:57.893111 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:57.893118 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:57.893346 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:57.893525 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:57.893859 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.894968 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.896527 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.897091 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.897352 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:57.897518 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:57.897653 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 06:37:57.897757 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":84,"threads_started":1}
I20260928 06:37:57.899075 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:57.891874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:32797 (local address 127.0.0.1:45190)
0928 06:37:57.893278 (+ 1404us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:57.893292 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:57.893377 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:57.894276 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:57.894279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:57.894304 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:57.894431 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:57.894437 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.896383 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.896390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:57.897593 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:57.897602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.898678 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.898681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:57.898761 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:57.898777 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 06:37:57.898825 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 06:37:57.898919 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1341,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260928 06:37:57.899124 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:57.893280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42184 (local address 127.5.22.3:34975)
0928 06:37:57.893843 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:57.893854 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:57.893876 (+ 22us) server_negotiation.cc:407] Connection header received
0928 06:37:57.893950 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:57.893955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:57.894124 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:57.894275 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:57.894679 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.895749 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.897485 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.898286 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.898435 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:57.898578 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:57.898888 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 06:37:57.898984 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":169,"threads_started":1}
I20260928 06:37:57.899312 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:57.892771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:34975 (local address 127.0.0.1:42184)
0928 06:37:57.893550 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:57.893562 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:57.893638 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:57.894483 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:57.894485 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:57.894501 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:57.894570 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:57.894574 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.895862 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.895865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:57.897327 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:57.897338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.898645 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.898649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:57.898738 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:57.898761 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 06:37:57.898840 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:37:57.898919 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":21,"thread_start_us":109,"threads_started":1}
I20260928 06:37:57.899971 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:57.892694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45190 (local address 127.5.22.2:32797)
0928 06:37:57.893863 (+ 1169us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:57.893875 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:57.893908 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:37:57.893971 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:57.893975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:57.894124 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:57.894275 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:57.894571 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.896234 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.897734 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.898537 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.899543 (+ 1006us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:57.899742 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:57.899784 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 06:37:57.899853 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1078,"thread_start_us":151,"threads_started":1}
I20260928 06:37:57.901890 19422 tablet_service.cc:1467] Processing CreateTablet for tablet fab743a6e8d04633bf503fe663b86515 (DEFAULT_TABLE table=test-table [id=ba8ecfe985a84b5d933b106529f44b35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:57.901827 19152 tablet_service.cc:1467] Processing CreateTablet for tablet fab743a6e8d04633bf503fe663b86515 (DEFAULT_TABLE table=test-table [id=ba8ecfe985a84b5d933b106529f44b35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:57.902165 19287 tablet_service.cc:1467] Processing CreateTablet for tablet fab743a6e8d04633bf503fe663b86515 (DEFAULT_TABLE table=test-table [id=ba8ecfe985a84b5d933b106529f44b35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:57.903043 19152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fab743a6e8d04633bf503fe663b86515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:57.903044 19422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fab743a6e8d04633bf503fe663b86515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:57.903272 19287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fab743a6e8d04633bf503fe663b86515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:57.909621 19506 tablet_bootstrap.cc:492] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513: Bootstrap starting.
I20260928 06:37:57.911093 19507 tablet_bootstrap.cc:492] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae: Bootstrap starting.
I20260928 06:37:57.911571 19508 tablet_bootstrap.cc:492] T fab743a6e8d04633bf503fe663b86515 P 0bc647d664234b94808b88315a39b22f: Bootstrap starting.
I20260928 06:37:57.912602 19506 tablet_bootstrap.cc:654] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:57.913345 19506 log.cc:824] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:57.913512 19508 tablet_bootstrap.cc:654] T fab743a6e8d04633bf503fe663b86515 P 0bc647d664234b94808b88315a39b22f: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:57.914728 19508 log.cc:824] T fab743a6e8d04633bf503fe663b86515 P 0bc647d664234b94808b88315a39b22f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:57.915665 19507 tablet_bootstrap.cc:654] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:57.915693 19506 tablet_bootstrap.cc:492] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513: No bootstrap required, opened a new log
I20260928 06:37:57.915884 19506 ts_tablet_manager.cc:1397] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 06:37:57.916493 19508 tablet_bootstrap.cc:492] T fab743a6e8d04633bf503fe663b86515 P 0bc647d664234b94808b88315a39b22f: No bootstrap required, opened a new log
I20260928 06:37:57.916710 19508 ts_tablet_manager.cc:1397] T fab743a6e8d04633bf503fe663b86515 P 0bc647d664234b94808b88315a39b22f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 06:37:57.916868 19507 log.cc:824] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:57.919008 19507 tablet_bootstrap.cc:492] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae: No bootstrap required, opened a new log
I20260928 06:37:57.919199 19507 ts_tablet_manager.cc:1397] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260928 06:37:57.919894 19506 raft_consensus.cc:348] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } }
I20260928 06:37:57.920156 19506 raft_consensus.cc:374] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:57.920228 19506 raft_consensus.cc:729] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8bcffaeb5a4a4eafe60c4a77c6f513, State: Initialized, Role: FOLLOWER
I20260928 06:37:57.920804 19506 consensus_queue.cc:260] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } }
I20260928 06:37:57.920680 19508 raft_consensus.cc:348] T fab743a6e8d04633bf503fe663b86515 P 0bc647d664234b94808b88315a39b22f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } }
I20260928 06:37:57.920960 19508 raft_consensus.cc:374] T fab743a6e8d04633bf503fe663b86515 P 0bc647d664234b94808b88315a39b22f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:57.921051 19508 raft_consensus.cc:729] T fab743a6e8d04633bf503fe663b86515 P 0bc647d664234b94808b88315a39b22f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc647d664234b94808b88315a39b22f, State: Initialized, Role: FOLLOWER
I20260928 06:37:57.921499 19508 consensus_queue.cc:260] T fab743a6e8d04633bf503fe663b86515 P 0bc647d664234b94808b88315a39b22f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } }
I20260928 06:37:57.921866 19506 ts_tablet_manager.cc:1428] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 06:37:57.922190 19354 heartbeater.cc:503] Master 127.5.22.62:39465 was elected leader, sending a full tablet report...
I20260928 06:37:57.922190 19489 heartbeater.cc:503] Master 127.5.22.62:39465 was elected leader, sending a full tablet report...
I20260928 06:37:57.922480 19508 ts_tablet_manager.cc:1428] T fab743a6e8d04633bf503fe663b86515 P 0bc647d664234b94808b88315a39b22f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 06:37:57.923009 19507 raft_consensus.cc:348] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } }
I20260928 06:37:57.923270 19507 raft_consensus.cc:374] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:57.923339 19507 raft_consensus.cc:729] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8afe0ccc5aaf41e68c49622dc18d96ae, State: Initialized, Role: FOLLOWER
I20260928 06:37:57.923774 19507 consensus_queue.cc:260] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } }
I20260928 06:37:57.924717 19507 ts_tablet_manager.cc:1428] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260928 06:37:57.924952 19219 heartbeater.cc:503] Master 127.5.22.62:39465 was elected leader, sending a full tablet report...
I20260928 06:37:57.978034 19512 raft_consensus.cc:482] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:37:57.978272 19512 raft_consensus.cc:504] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } }
I20260928 06:37:57.979530 19512 leader_election.cc:288] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8afe0ccc5aaf41e68c49622dc18d96ae (127.5.22.1:40803), 0bc647d664234b94808b88315a39b22f (127.5.22.3:34975)
I20260928 06:37:57.983485 19356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:57.979717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:40803 (local address 127.5.22.2:42857)
0928 06:37:57.979924 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:57.979940 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:57.980042 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:57.980373 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:57.980376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:57.980395 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:57.980532 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:57.980541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.981409 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.981415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:57.982304 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:57.982338 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.982983 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.982990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:57.983103 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:57.983129 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 06:37:57.983226 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 06:37:57.983324 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":62}
I20260928 06:37:57.983484 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:57.979800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:42857 (local address 127.5.22.1:40803)
0928 06:37:57.979938 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:57.979943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:57.979987 (+ 44us) server_negotiation.cc:407] Connection header received
0928 06:37:57.980184 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:57.980188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:57.980257 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:57.980336 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:57.980650 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.981296 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.982444 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.982858 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.982948 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:57.983028 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:57.983247 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 06:37:57.983343 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:37:57.983522 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:57.979787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:34975 (local address 127.5.22.2:57829)
0928 06:37:57.980169 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:57.980183 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:57.980282 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:57.980579 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:57.980581 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:57.980617 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:57.980730 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:57.980736 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.981459 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.981462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:57.982374 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:57.982387 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.983074 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.983080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:57.983172 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:57.983217 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 06:37:57.983298 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 06:37:57.983368 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":32,"thread_start_us":156,"threads_started":1}
I20260928 06:37:57.983532 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:57.979973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:57829 (local address 127.5.22.3:34975)
0928 06:37:57.980114 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:57.980120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:57.980267 (+ 147us) server_negotiation.cc:407] Connection header received
0928 06:37:57.980416 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:57.980419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:57.980470 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:57.980556 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:57.980842 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.981307 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.982513 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:57.982949 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:57.983050 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:57.983113 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:57.983342 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 06:37:57.983424 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:37:57.984328 19172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fab743a6e8d04633bf503fe663b86515" candidate_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" is_pre_election: true
I20260928 06:37:57.984370 19442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fab743a6e8d04633bf503fe663b86515" candidate_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc647d664234b94808b88315a39b22f" is_pre_election: true
I20260928 06:37:57.984658 19172 raft_consensus.cc:2454] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba8bcffaeb5a4a4eafe60c4a77c6f513 in term 0.
I20260928 06:37:57.984683 19442 raft_consensus.cc:2454] T fab743a6e8d04633bf503fe663b86515 P 0bc647d664234b94808b88315a39b22f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba8bcffaeb5a4a4eafe60c4a77c6f513 in term 0.
I20260928 06:37:57.985227 19240 leader_election.cc:302] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0bc647d664234b94808b88315a39b22f, 8afe0ccc5aaf41e68c49622dc18d96ae, ba8bcffaeb5a4a4eafe60c4a77c6f513; no voters:
I20260928 06:37:57.985535 19512 raft_consensus.cc:2781] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:37:57.985616 19512 raft_consensus.cc:482] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:37:57.985689 19512 raft_consensus.cc:3037] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:57.986917 19512 raft_consensus.cc:504] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } }
I20260928 06:37:57.987219 19512 leader_election.cc:288] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [CANDIDATE]: Term 1 election: Requested vote from peers 8afe0ccc5aaf41e68c49622dc18d96ae (127.5.22.1:40803), 0bc647d664234b94808b88315a39b22f (127.5.22.3:34975)
I20260928 06:37:57.987622 19172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fab743a6e8d04633bf503fe663b86515" candidate_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae"
I20260928 06:37:57.987622 19442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fab743a6e8d04633bf503fe663b86515" candidate_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc647d664234b94808b88315a39b22f"
I20260928 06:37:57.987746 19442 raft_consensus.cc:3037] T fab743a6e8d04633bf503fe663b86515 P 0bc647d664234b94808b88315a39b22f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:57.987747 19172 raft_consensus.cc:3037] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:57.988991 19172 raft_consensus.cc:2454] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba8bcffaeb5a4a4eafe60c4a77c6f513 in term 1.
I20260928 06:37:57.988991 19442 raft_consensus.cc:2454] T fab743a6e8d04633bf503fe663b86515 P 0bc647d664234b94808b88315a39b22f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba8bcffaeb5a4a4eafe60c4a77c6f513 in term 1.
I20260928 06:37:57.989287 19240 leader_election.cc:302] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8afe0ccc5aaf41e68c49622dc18d96ae, ba8bcffaeb5a4a4eafe60c4a77c6f513; no voters:
I20260928 06:37:57.989481 19512 raft_consensus.cc:2781] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:57.989704 19512 raft_consensus.cc:686] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [term 1 LEADER]: Becoming Leader. State: Replica: ba8bcffaeb5a4a4eafe60c4a77c6f513, State: Running, Role: LEADER
I20260928 06:37:57.990048 19512 consensus_queue.cc:237] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } }
I20260928 06:37:57.993460 19025 catalog_manager.cc:5495] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba8bcffaeb5a4a4eafe60c4a77c6f513 (127.5.22.2). New cstate: current_term: 1 leader_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260928 06:37:58.024525 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:58.020830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45206 (local address 127.5.22.2:32797)
0928 06:37:58.020977 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:58.020982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:58.020999 (+ 17us) server_negotiation.cc:407] Connection header received
0928 06:37:58.021096 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:58.021099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:58.021154 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:58.021272 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:37:58.021547 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.022051 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.023037 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.023258 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.023366 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:58.023680 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:37:58.024095 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:37:58.024230 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:58.024300 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 06:37:58.024357 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 06:37:58.030834 19442 raft_consensus.cc:1261] T fab743a6e8d04633bf503fe663b86515 P 0bc647d664234b94808b88315a39b22f [term 1 FOLLOWER]: Refusing update from remote peer ba8bcffaeb5a4a4eafe60c4a77c6f513: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:58.030839 19172 raft_consensus.cc:1261] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Refusing update from remote peer ba8bcffaeb5a4a4eafe60c4a77c6f513: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:58.031697 19512 consensus_queue.cc:1035] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:58.031942 19516 consensus_queue.cc:1035] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:58.048943 19024 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:37:58.049105 19024 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:37:58.051658 19024 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63: Sending DeleteTablet for 3 replicas of tablet fab743a6e8d04633bf503fe663b86515
I20260928 06:37:58.052485 19152 tablet_service.cc:1514] Processing DeleteTablet for tablet fab743a6e8d04633bf503fe663b86515 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:58 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:41034
I20260928 06:37:58.052516 19287 tablet_service.cc:1514] Processing DeleteTablet for tablet fab743a6e8d04633bf503fe663b86515 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:58 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:45190
I20260928 06:37:58.052682 19422 tablet_service.cc:1514] Processing DeleteTablet for tablet fab743a6e8d04633bf503fe663b86515 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:58 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:42184
I20260928 06:37:58.053082 19529 tablet_replica.cc:330] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513: stopping tablet replica
I20260928 06:37:58.053481 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 19089
I20260928 06:37:58.053479 19530 tablet_replica.cc:330] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae: stopping tablet replica
I20260928 06:37:58.053493 19529 raft_consensus.cc:2229] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:37:58.053790 19530 raft_consensus.cc:2229] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:37:58.053875 19529 raft_consensus.cc:2258] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:58.054363 19531 tablet_replica.cc:330] T fab743a6e8d04633bf503fe663b86515 P 0bc647d664234b94808b88315a39b22f: stopping tablet replica
I20260928 06:37:58.054664 19531 raft_consensus.cc:2229] T fab743a6e8d04633bf503fe663b86515 P 0bc647d664234b94808b88315a39b22f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:37:58.054922 19531 raft_consensus.cc:2258] T fab743a6e8d04633bf503fe663b86515 P 0bc647d664234b94808b88315a39b22f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:58.055680 19529 ts_tablet_manager.cc:1905] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:37:58.057255 19531 ts_tablet_manager.cc:1905] T fab743a6e8d04633bf503fe663b86515 P 0bc647d664234b94808b88315a39b22f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:37:58.058617 19529 ts_tablet_manager.cc:1918] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:37:58.058758 19529 log.cc:1196] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/wal/wals/fab743a6e8d04633bf503fe663b86515
I20260928 06:37:58.059150 19529 ts_tablet_manager.cc:1939] T fab743a6e8d04633bf503fe663b86515 P ba8bcffaeb5a4a4eafe60c4a77c6f513: Deleting consensus metadata
I20260928 06:37:58.060468 19012 catalog_manager.cc:4841] TS ba8bcffaeb5a4a4eafe60c4a77c6f513 (127.5.22.2:32797): tablet fab743a6e8d04633bf503fe663b86515 (table test-table [id=ba8ecfe985a84b5d933b106529f44b35]) successfully deleted
I20260928 06:37:58.060719 19531 ts_tablet_manager.cc:1918] T fab743a6e8d04633bf503fe663b86515 P 0bc647d664234b94808b88315a39b22f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:37:58.060825 19531 log.cc:1196] T fab743a6e8d04633bf503fe663b86515 P 0bc647d664234b94808b88315a39b22f: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/fab743a6e8d04633bf503fe663b86515
I20260928 06:37:58.061190 19531 ts_tablet_manager.cc:1939] T fab743a6e8d04633bf503fe663b86515 P 0bc647d664234b94808b88315a39b22f: Deleting consensus metadata
I20260928 06:37:58.062527 19011 catalog_manager.cc:4841] TS 0bc647d664234b94808b88315a39b22f (127.5.22.3:34975): tablet fab743a6e8d04633bf503fe663b86515 (table test-table [id=ba8ecfe985a84b5d933b106529f44b35]) successfully deleted
W20260928 06:37:58.063084 19010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.22.1:40803 (error 108)
I20260928 06:37:58.063618 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 19224
I20260928 06:37:58.064510 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:58.063885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:40803 (local address 127.0.0.1:41048)
0928 06:37:58.064324 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:58.064406 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.1:40803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337}
W20260928 06:37:58.064843 19010 catalog_manager.cc:4584] TS 8afe0ccc5aaf41e68c49622dc18d96ae (127.5.22.1:40803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fab743a6e8d04633bf503fe663b86515: Network error: Client connection negotiation failed: client connection to 127.5.22.1:40803: connect: Connection refused (error 111)
I20260928 06:37:58.070420 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 19359
I20260928 06:37:58.076495 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 18994
I20260928 06:37:58.083271 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:39465
--webserver_interface=127.5.22.62
--webserver_port=45167
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:41197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:39465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:58.195870 19533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:58.196172 19533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:58.196215 19533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:58.200382 19533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:58.200456 19533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:58.200476 19533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:58.200493 19533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:58.200510 19533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:58.205998 19533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:41197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:39465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:39465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=45167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:58.207231 19533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:58.208321 19533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:58.224859 19542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:58.224958 19540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:58.224938 19539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:58.225134 19533 server_base.cc:1034] running on GCE node
I20260928 06:37:58.225899 19533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:58.227156 19533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:58.228331 19533 hybrid_clock.cc:648] HybridClock initialized: now 1790577478228311 us; error 35 us; skew 500 ppm
Sep 28 06:37:58 dist-test-slave-j0qf krb5kdc[18980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577478, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:58.231310 19533 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:37:58.232282 19533 webserver.cc:460] Webserver started at http://127.5.22.62:45167/ using document root <none> and password file <none>
I20260928 06:37:58.232832 19533 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:58.232905 19533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:58.236984 19533 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:37:58.239234 19549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:58.240633 19533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:37:58.240778 19533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "34e41488fd0749dd8517addf86daae63"
format_stamp: "Formatted at 2026-09-28 06:37:57 on dist-test-slave-j0qf"
server_key: "8f751842537442d359b56869617f1c54"
server_key_iv: "f6219250bb9897a38249920182d79756"
server_key_version: "encryptionkey@0"
I20260928 06:37:58.241417 19533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:58.262876 19533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:58.265985 19533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:58.266198 19533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:58.274173 19533 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:39465
I20260928 06:37:58.274204 19601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:39465 every 8 connection(s)
I20260928 06:37:58.275408 19533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:58.279804 19602 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:37:58.280740 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 19533
I20260928 06:37:58.281236 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:40803
--local_ip_for_outbound_sockets=127.5.22.1
--tserver_master_addrs=127.5.22.62:39465
--webserver_port=45035
--webserver_interface=127.5.22.1
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:41197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:37:58.281873 19602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63: Bootstrap starting.
I20260928 06:37:58.300004 19602 log.cc:824] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:58.308897 19602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:37:58.309284 19602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63: Bootstrap complete.
I20260928 06:37:58.313645 19602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e41488fd0749dd8517addf86daae63" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 39465 } }
I20260928 06:37:58.314054 19602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e41488fd0749dd8517addf86daae63, State: Initialized, Role: FOLLOWER
I20260928 06:37:58.314591 19602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e41488fd0749dd8517addf86daae63" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 39465 } }
I20260928 06:37:58.314707 19602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:58.314796 19602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:58.314920 19602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:37:58.316929 19602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e41488fd0749dd8517addf86daae63" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 39465 } }
I20260928 06:37:58.317266 19602 leader_election.cc:302] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34e41488fd0749dd8517addf86daae63; no voters:
I20260928 06:37:58.317605 19602 leader_election.cc:288] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:37:58.317724 19606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:37:58.317953 19606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [term 2 LEADER]: Becoming Leader. State: Replica: 34e41488fd0749dd8517addf86daae63, State: Running, Role: LEADER
I20260928 06:37:58.318254 19606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e41488fd0749dd8517addf86daae63" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 39465 } }
I20260928 06:37:58.318919 19602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:58.319252 19608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34e41488fd0749dd8517addf86daae63. Latest consensus state: current_term: 2 leader_uuid: "34e41488fd0749dd8517addf86daae63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e41488fd0749dd8517addf86daae63" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 39465 } } }
I20260928 06:37:58.319422 19608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:58.319613 19607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "34e41488fd0749dd8517addf86daae63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e41488fd0749dd8517addf86daae63" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 39465 } } }
I20260928 06:37:58.319725 19607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:58.320222 19610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:58.325100 19610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:58.325852 19610 catalog_manager.cc:1259] Loaded cluster ID: 6c0d22fc78814b4da244d4b25fd051f8
I20260928 06:37:58.325918 19610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:58.327319 19610 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 06:37:58.332216 19610 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 06:37:58.337925 19610 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:58.338759 19610 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63: Loaded TSK: 0
I20260928 06:37:58.339973 19610 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:37:58.398510 19604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:58.398814 19604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:58.398901 19604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:58.402995 19604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:58.403080 19604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:58.403167 19604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:37:58.408743 19604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:41197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:40803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=45035
--tserver_master_addrs=127.5.22.62:39465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:58.409855 19604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:58.410967 19604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:58.422150 19632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:58.422206 19629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:58.422411 19604 server_base.cc:1034] running on GCE node
W20260928 06:37:58.422529 19630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:58.423014 19604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:58.423722 19604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:58.424909 19604 hybrid_clock.cc:648] HybridClock initialized: now 1790577478424869 us; error 65 us; skew 500 ppm
Sep 28 06:37:58 dist-test-slave-j0qf krb5kdc[18980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577478, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:58.427878 19604 init.cc:377] Logged in from keytab as kudu/127.5.22.1@KRBTEST.COM (short username kudu)
I20260928 06:37:58.428884 19604 webserver.cc:460] Webserver started at http://127.5.22.1:45035/ using document root <none> and password file <none>
I20260928 06:37:58.429455 19604 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:58.429562 19604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:58.433625 19604 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:37:58.435884 19639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:58.437222 19604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:37:58.437317 19604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "8afe0ccc5aaf41e68c49622dc18d96ae"
format_stamp: "Formatted at 2026-09-28 06:37:57 on dist-test-slave-j0qf"
server_key: "6189a654d4afaa2a89fc2da02c880f2c"
server_key_iv: "393a0a9c61976d80018652eb2fa3e9ad"
server_key_version: "encryptionkey@0"
I20260928 06:37:58.437965 19604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:58.456952 19604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:58.460487 19604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:58.460700 19604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:58.461385 19604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:58.462697 19646 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:37:58.468058 19604 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:37:58.468155 19604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260928 06:37:58.468240 19604 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:37:58.470009 19604 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:37:58.470084 19604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 06:37:58.470436 19646 tablet_bootstrap.cc:492] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae: Bootstrap starting.
I20260928 06:37:58.481259 19604 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:40803
I20260928 06:37:58.481315 19755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:40803 every 8 connection(s)
I20260928 06:37:58.482247 19604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:37:58.488669 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 19604
I20260928 06:37:58.489068 19646 log.cc:824] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:58.489171 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:32797
--local_ip_for_outbound_sockets=127.5.22.2
--tserver_master_addrs=127.5.22.62:39465
--webserver_port=40281
--webserver_interface=127.5.22.2
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:41197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:37:58 dist-test-slave-j0qf krb5kdc[18980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577478, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:37:58.496968 19646 tablet_bootstrap.cc:492] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:37:58.497354 19646 tablet_bootstrap.cc:492] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae: Bootstrap complete.
I20260928 06:37:58.498651 19646 ts_tablet_manager.cc:1397] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae: Time spent bootstrapping tablet: real 0.028s user 0.024s sys 0.004s
I20260928 06:37:58.501461 19646 raft_consensus.cc:348] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } }
I20260928 06:37:58.501988 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:58.484640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:38809 (local address 127.5.22.62:39465)
0928 06:37:58.486381 (+ 1741us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:58.486398 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:58.486439 (+ 41us) server_negotiation.cc:407] Connection header received
0928 06:37:58.486651 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:58.486661 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:58.486958 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:58.487276 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:58.488436 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.489697 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.491566 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.492250 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.492459 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:58.495652 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:58.495661 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:58.495682 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:58.495725 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:58.498094 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:58.498817 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:58.498823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:58.498830 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:58.498985 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:58.499457 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:58.499460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:58.499462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:58.499906 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:58.500038 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:58.500776 (+ 738us) server_negotiation.cc:299] Negotiation successful
0928 06:37:58.501088 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1590,"thread_start_us":148,"threads_started":1}
I20260928 06:37:58.502002 19646 raft_consensus.cc:729] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8afe0ccc5aaf41e68c49622dc18d96ae, State: Initialized, Role: FOLLOWER
I20260928 06:37:58.502506 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:58.484602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.1:38809)
0928 06:37:58.485083 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:58.485138 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:58.485963 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:58.487328 (+ 1365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:58.487336 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:58.487763 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:58.488213 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:58.488228 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.489878 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.489885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:58.491417 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:58.491432 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.492404 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.492409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:58.492643 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:58.493404 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:58.493429 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:58.495426 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:58.498233 (+ 2807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:58.498241 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:58.498257 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:58.498652 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:58.499101 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:58.499108 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:58.499111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:58.499314 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:58.500043 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:58.500052 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:58.500393 (+ 341us) client_negotiation.cc:769] Sending connection context
0928 06:37:58.500618 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 06:37:58.501012 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":139,"threads_started":1}
I20260928 06:37:58.502511 19646 consensus_queue.cc:260] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } }
I20260928 06:37:58.503511 19646 ts_tablet_manager.cc:1428] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260928 06:37:58.504797 19756 heartbeater.cc:347] Connected to a master server at 127.5.22.62:39465
I20260928 06:37:58.505033 19756 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:58.505554 19756 heartbeater.cc:511] Master 127.5.22.62:39465 requested a full tablet report, sending...
I20260928 06:37:58.507967 19565 ts_manager.cc:194] Registered new tserver with Master: 8afe0ccc5aaf41e68c49622dc18d96ae (127.5.22.1:40803)
I20260928 06:37:58.511303 19565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.1@KRBTEST.COM'} at 127.5.22.1:38809
I20260928 06:37:58.513748 19756 heartbeater.cc:503] Master 127.5.22.62:39465 was elected leader, sending a full tablet report...
Sep 28 06:37:58 dist-test-slave-j0qf krb5kdc[18980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577478, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for kudu/127.5.22.1@KRBTEST.COM
I20260928 06:37:58.522147 19765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:58.510741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55348 (local address 127.5.22.1:40803)
0928 06:37:58.511126 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:58.511138 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:58.511548 (+ 410us) server_negotiation.cc:407] Connection header received
0928 06:37:58.511737 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:58.511742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:58.511934 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:58.512083 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:58.512926 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.513936 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.515187 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.515667 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.515761 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:58.518469 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:58.518477 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:58.518486 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:58.518506 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:58.520187 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:58.520771 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:58.520776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:58.520777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:58.520851 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:58.521156 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:58.521158 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:58.521159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:58.521376 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:58.521478 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:58.521887 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 06:37:58.522021 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":125,"threads_started":1}
I20260928 06:37:58.522189 19764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:58.510638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:40803 (local address 127.0.0.1:55348)
0928 06:37:58.511400 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:58.511450 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:58.511591 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:58.512209 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:58.512216 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:58.512653 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:58.512783 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:58.512791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.514062 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.514068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:58.515043 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:58.515055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.515920 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.515937 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:58.516046 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:58.516592 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:58.516607 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:58.518349 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:58.520310 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:58.520319 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:58.520324 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:58.520644 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:58.520967 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:58.520970 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:58.520972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:58.521071 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:58.521490 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:58.521496 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:58.521626 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 06:37:58.521798 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 06:37:58.522024 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":118,"threads_started":1}
I20260928 06:37:58.523286 19690 tablet_service.cc:1514] Processing DeleteTablet for tablet fab743a6e8d04633bf503fe663b86515 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:58 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:55348
I20260928 06:37:58.524168 19766 tablet_replica.cc:330] stopping tablet replica
I20260928 06:37:58.524497 19766 raft_consensus.cc:2229] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:37:58.524757 19766 raft_consensus.cc:2258] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:58.525884 19766 ts_tablet_manager.cc:1905] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:37:58.530642 19766 ts_tablet_manager.cc:1918] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:37:58.530748 19766 log.cc:1196] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/wal/wals/fab743a6e8d04633bf503fe663b86515
I20260928 06:37:58.531049 19766 ts_tablet_manager.cc:1939] T fab743a6e8d04633bf503fe663b86515 P 8afe0ccc5aaf41e68c49622dc18d96ae: Deleting consensus metadata
I20260928 06:37:58.533108 19550 catalog_manager.cc:4841] TS 8afe0ccc5aaf41e68c49622dc18d96ae (127.5.22.1:40803): tablet fab743a6e8d04633bf503fe663b86515 (table test-table [id=ba8ecfe985a84b5d933b106529f44b35]) successfully deleted
W20260928 06:37:58.612190 19762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:58.612463 19762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:58.612505 19762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:58.616371 19762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:58.616421 19762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:58.616477 19762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:37:58.621896 19762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:41197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:32797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=40281
--tserver_master_addrs=127.5.22.62:39465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:58.623090 19762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:58.624048 19762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:58.633968 19771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:58.633992 19772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:58.633984 19774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:58.634133 19762 server_base.cc:1034] running on GCE node
I20260928 06:37:58.634719 19762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:58.635406 19762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:58.636590 19762 hybrid_clock.cc:648] HybridClock initialized: now 1790577478636562 us; error 49 us; skew 500 ppm
Sep 28 06:37:58 dist-test-slave-j0qf krb5kdc[18980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577478, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:58.639423 19762 init.cc:377] Logged in from keytab as kudu/127.5.22.2@KRBTEST.COM (short username kudu)
I20260928 06:37:58.640268 19762 webserver.cc:460] Webserver started at http://127.5.22.2:40281/ using document root <none> and password file <none>
I20260928 06:37:58.640853 19762 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:58.640952 19762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:58.645002 19762 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 06:37:58.647307 19781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:58.648613 19762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:37:58.648712 19762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513"
format_stamp: "Formatted at 2026-09-28 06:37:57 on dist-test-slave-j0qf"
server_key: "dd1a1d571a7462c99551d426c968baac"
server_key_iv: "08f6c7a64c0290f9de31ce4774d8f82c"
server_key_version: "encryptionkey@0"
I20260928 06:37:58.649241 19762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:58.673280 19762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:58.676498 19762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:58.676724 19762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:58.677299 19762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:58.678093 19762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:58.678140 19762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:58.678180 19762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:58.678195 19762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:58.687414 19762 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:32797
I20260928 06:37:58.687558 19895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:32797 every 8 connection(s)
I20260928 06:37:58.688465 19762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:37:58.697118 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 19762
I20260928 06:37:58.697587 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:34975
--local_ip_for_outbound_sockets=127.5.22.3
--tserver_master_addrs=127.5.22.62:39465
--webserver_port=42499
--webserver_interface=127.5.22.3
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:41197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:37:58 dist-test-slave-j0qf krb5kdc[18980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577478, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:37:58.703469 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:58.690455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:46915 (local address 127.5.22.62:39465)
0928 06:37:58.690644 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:58.690648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:58.691458 (+ 810us) server_negotiation.cc:407] Connection header received
0928 06:37:58.692756 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:58.692759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:58.692813 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:58.692890 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:58.694073 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.694723 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.695924 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.696145 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.696225 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:58.699430 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:58.699437 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:58.699442 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:58.699480 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:58.701061 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:58.701618 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:58.701624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:58.701629 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:58.701725 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:58.702105 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:58.702111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:58.702115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:58.702411 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:58.702523 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:58.703059 (+ 536us) server_negotiation.cc:299] Negotiation successful
0928 06:37:58.703221 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 06:37:58.704258 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:58.690811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.2:46915)
0928 06:37:58.691286 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:58.691336 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:58.692539 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:58.693053 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:58.693059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:58.693425 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:58.693928 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:58.693943 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.694857 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.694860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:58.695807 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:58.695817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.696329 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.696333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:58.696495 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:58.697266 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:58.697293 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:58.699216 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:58.701182 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:58.701188 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:58.701202 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:58.701507 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:58.701847 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:58.701850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:58.701852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:58.701997 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:58.702526 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:58.702531 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:58.702800 (+ 269us) client_negotiation.cc:769] Sending connection context
0928 06:37:58.702972 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 06:37:58.703247 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":120,"threads_started":1}
I20260928 06:37:58.705507 19896 heartbeater.cc:347] Connected to a master server at 127.5.22.62:39465
I20260928 06:37:58.705791 19896 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:58.706497 19896 heartbeater.cc:511] Master 127.5.22.62:39465 requested a full tablet report, sending...
I20260928 06:37:58.707790 19565 ts_manager.cc:194] Registered new tserver with Master: ba8bcffaeb5a4a4eafe60c4a77c6f513 (127.5.22.2:32797)
I20260928 06:37:58.708429 19565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.2@KRBTEST.COM'} at 127.5.22.2:46915
W20260928 06:37:58.816391 19899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:58.816651 19899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:58.816694 19899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:58.821463 19899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:58.821520 19899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:58.821605 19899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:37:58.827069 19899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:41197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:34975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=42499
--tserver_master_addrs=127.5.22.62:39465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:58.828090 19899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:58.829101 19899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:58.839226 19907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:58.839226 19904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:58.839232 19905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:58.839620 19899 server_base.cc:1034] running on GCE node
I20260928 06:37:58.840070 19899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:58.840751 19899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:58.842012 19899 hybrid_clock.cc:648] HybridClock initialized: now 1790577478841976 us; error 46 us; skew 500 ppm
Sep 28 06:37:58 dist-test-slave-j0qf krb5kdc[18980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577478, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:58.845304 19899 init.cc:377] Logged in from keytab as kudu/127.5.22.3@KRBTEST.COM (short username kudu)
I20260928 06:37:58.846275 19899 webserver.cc:460] Webserver started at http://127.5.22.3:42499/ using document root <none> and password file <none>
I20260928 06:37:58.846915 19899 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:58.846997 19899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:58.851049 19899 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:37:58.853268 19914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:58.854754 19899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:37:58.854900 19899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "0bc647d664234b94808b88315a39b22f"
format_stamp: "Formatted at 2026-09-28 06:37:57 on dist-test-slave-j0qf"
server_key: "40911fae3b63f3fe72cd46c8e1047756"
server_key_iv: "e73d379fcd449c1e54699268b6697522"
server_key_version: "encryptionkey@0"
I20260928 06:37:58.855576 19899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:58.881268 19899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:58.884927 19899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:58.885151 19899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:58.885875 19899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:58.886775 19899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:58.886854 19899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:58.886940 19899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:58.886989 19899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:58.896579 19899 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:34975
I20260928 06:37:58.896600 20028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:34975 every 8 connection(s)
I20260928 06:37:58.897639 19899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:37:58.906250 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 19899
Sep 28 06:37:58 dist-test-slave-j0qf krb5kdc[18980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577478, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:37:58.912922 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:58.899601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:52475 (local address 127.5.22.62:39465)
0928 06:37:58.899780 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:58.899784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:58.900667 (+ 883us) server_negotiation.cc:407] Connection header received
0928 06:37:58.902066 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:58.902069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:58.902117 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:58.902198 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:58.903431 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.903979 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.905242 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.905500 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.905585 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:58.908698 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:58.908702 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:58.908706 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:58.908737 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:58.910456 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:58.911020 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:58.911023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:58.911025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:58.911118 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:58.911504 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:58.911509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:58.911512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:58.911714 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:58.911799 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:58.912549 (+ 750us) server_negotiation.cc:299] Negotiation successful
0928 06:37:58.912682 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 06:37:58.913815 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:58.900004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.3:52475)
0928 06:37:58.900499 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:58.900545 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:58.901809 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:58.902402 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:58.902409 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:58.902776 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:58.903273 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:58.903286 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.904091 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.904093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:58.905116 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:58.905125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.905665 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.905669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:58.905819 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:58.906590 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:58.906609 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:58.908505 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:58.910578 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:58.910584 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:58.910602 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:58.910877 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:58.911220 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:58.911225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:58.911230 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:58.911392 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:58.911832 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:58.911839 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:58.912219 (+ 380us) client_negotiation.cc:769] Sending connection context
0928 06:37:58.912468 (+ 249us) client_negotiation.cc:240] Negotiation successful
0928 06:37:58.912765 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":125,"threads_started":1}
I20260928 06:37:58.915329 20029 heartbeater.cc:347] Connected to a master server at 127.5.22.62:39465
I20260928 06:37:58.915634 20029 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:58.916261 20029 heartbeater.cc:511] Master 127.5.22.62:39465 requested a full tablet report, sending...
I20260928 06:37:58.916298 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:58.907102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36382 (local address 127.5.22.62:39465)
0928 06:37:58.907436 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:58.907439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:58.907560 (+ 121us) server_negotiation.cc:407] Connection header received
0928 06:37:58.907689 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:58.907692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:58.907732 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:58.907811 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:58.908693 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.909589 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.910618 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.910857 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.910947 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:58.912564 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:58.912567 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:58.912570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:58.912591 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:58.914477 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:58.914964 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:58.914967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:58.914968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:58.915053 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:58.915354 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:58.915357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:58.915359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:58.915579 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:58.915812 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:58.916000 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 06:37:58.916127 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":107,"threads_started":1}
I20260928 06:37:58.917469 19565 ts_manager.cc:194] Registered new tserver with Master: 0bc647d664234b94808b88315a39b22f (127.5.22.3:34975)
I20260928 06:37:58.918167 19565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.3@KRBTEST.COM'} at 127.5.22.3:52475
I20260928 06:37:58.918253 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:37:58.922983 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:58.919268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36390 (local address 127.5.22.62:39465)
0928 06:37:58.919410 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:58.919414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:58.919426 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:37:58.919495 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:58.919497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:58.919577 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:58.919684 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:37:58.920006 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.920689 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.921560 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.921820 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.921914 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:58.922158 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:37:58.922579 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:37:58.922726 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:58.922816 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 06:37:58.922867 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:37:58.923717 19565 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:36390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:37:58.927104 19565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:37:58.935266 19690 tablet_service.cc:1467] Processing CreateTablet for tablet 12e8c8877cd6418b98a9073b780bf8b0 (DEFAULT_TABLE table=test-table [id=66ae5b1261644cb5b2784e03ba40efc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:58.936022 19690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12e8c8877cd6418b98a9073b780bf8b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:58.938968 19646 tablet_bootstrap.cc:492] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae: Bootstrap starting.
I20260928 06:37:58.939874 19646 tablet_bootstrap.cc:654] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:58.940505 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:58.934322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49126 (local address 127.5.22.3:34975)
0928 06:37:58.934936 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:58.934949 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:58.934975 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:37:58.935062 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:58.935066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:58.935237 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:58.935408 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:58.935759 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.936964 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.939069 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.939832 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.939931 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:58.940107 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:58.940305 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 06:37:58.940384 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":169,"threads_started":1}
I20260928 06:37:58.940554 19764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:58.933975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:34975 (local address 127.0.0.1:49126)
0928 06:37:58.934165 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:58.934182 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:58.934266 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:58.935387 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:58.935390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:58.935408 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:58.935534 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:58.935541 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.937105 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.937109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:58.938787 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:58.938795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.939976 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.939980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:58.940112 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:58.940166 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 06:37:58.940278 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 06:37:58.940400 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":249}
I20260928 06:37:58.941581 19646 tablet_bootstrap.cc:492] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae: No bootstrap required, opened a new log
I20260928 06:37:58.941663 19646 ts_tablet_manager.cc:1397] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:37:58.942178 19646 raft_consensus.cc:348] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } }
I20260928 06:37:58.942350 19646 raft_consensus.cc:374] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:58.942420 19646 raft_consensus.cc:729] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8afe0ccc5aaf41e68c49622dc18d96ae, State: Initialized, Role: FOLLOWER
I20260928 06:37:58.942565 19646 consensus_queue.cc:260] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } }
I20260928 06:37:58.942998 19646 ts_tablet_manager.cc:1428] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:37:58.943715 19962 tablet_service.cc:1467] Processing CreateTablet for tablet 12e8c8877cd6418b98a9073b780bf8b0 (DEFAULT_TABLE table=test-table [id=66ae5b1261644cb5b2784e03ba40efc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:58.944393 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:58.934181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:32797 (local address 127.0.0.1:39172)
0928 06:37:58.934580 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:58.934595 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:58.934688 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:58.935756 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:58.935759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:58.935779 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:58.935869 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:58.935875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.937258 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.937261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:58.938661 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:58.938673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.943973 (+ 5300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.943977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:58.944072 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:58.944096 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 06:37:58.944181 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 06:37:58.944260 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":77,"thread_start_us":144,"threads_started":1}
I20260928 06:37:58.944792 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:58.934691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39172 (local address 127.5.22.2:32797)
0928 06:37:58.935068 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:58.935081 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:58.935107 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:37:58.935186 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:58.935191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:58.935413 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:58.935787 (+ 374us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:58.935969 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.937106 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.938804 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.943802 (+ 4998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.943945 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:58.944090 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:58.944560 (+ 470us) server_negotiation.cc:299] Negotiation successful
0928 06:37:58.944655 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":134,"threads_started":1}
I20260928 06:37:58.944826 19962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12e8c8877cd6418b98a9073b780bf8b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:58.947605 19829 tablet_service.cc:1467] Processing CreateTablet for tablet 12e8c8877cd6418b98a9073b780bf8b0 (DEFAULT_TABLE table=test-table [id=66ae5b1261644cb5b2784e03ba40efc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:58.948457 19829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12e8c8877cd6418b98a9073b780bf8b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:58.951953 20042 tablet_bootstrap.cc:492] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f: Bootstrap starting.
I20260928 06:37:58.954051 20042 tablet_bootstrap.cc:654] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:58.955150 20042 log.cc:824] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:58.955291 20043 tablet_bootstrap.cc:492] T 12e8c8877cd6418b98a9073b780bf8b0 P ba8bcffaeb5a4a4eafe60c4a77c6f513: Bootstrap starting.
I20260928 06:37:58.957060 20042 tablet_bootstrap.cc:492] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f: No bootstrap required, opened a new log
I20260928 06:37:58.957267 20042 ts_tablet_manager.cc:1397] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 06:37:58.957937 20043 tablet_bootstrap.cc:654] T 12e8c8877cd6418b98a9073b780bf8b0 P ba8bcffaeb5a4a4eafe60c4a77c6f513: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:58.958721 20043 log.cc:824] T 12e8c8877cd6418b98a9073b780bf8b0 P ba8bcffaeb5a4a4eafe60c4a77c6f513: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:58.960534 20043 tablet_bootstrap.cc:492] T 12e8c8877cd6418b98a9073b780bf8b0 P ba8bcffaeb5a4a4eafe60c4a77c6f513: No bootstrap required, opened a new log
I20260928 06:37:58.960510 20042 raft_consensus.cc:348] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } }
I20260928 06:37:58.960784 20043 ts_tablet_manager.cc:1397] T 12e8c8877cd6418b98a9073b780bf8b0 P ba8bcffaeb5a4a4eafe60c4a77c6f513: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 06:37:58.960811 20042 raft_consensus.cc:374] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:58.960870 20042 raft_consensus.cc:729] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc647d664234b94808b88315a39b22f, State: Initialized, Role: FOLLOWER
I20260928 06:37:58.961306 20042 consensus_queue.cc:260] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } }
I20260928 06:37:58.961982 20029 heartbeater.cc:503] Master 127.5.22.62:39465 was elected leader, sending a full tablet report...
I20260928 06:37:58.962107 20042 ts_tablet_manager.cc:1428] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 06:37:58.964398 20043 raft_consensus.cc:348] T 12e8c8877cd6418b98a9073b780bf8b0 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } }
I20260928 06:37:58.964720 20043 raft_consensus.cc:374] T 12e8c8877cd6418b98a9073b780bf8b0 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:58.964792 20043 raft_consensus.cc:729] T 12e8c8877cd6418b98a9073b780bf8b0 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8bcffaeb5a4a4eafe60c4a77c6f513, State: Initialized, Role: FOLLOWER
I20260928 06:37:58.965246 20043 consensus_queue.cc:260] T 12e8c8877cd6418b98a9073b780bf8b0 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } }
I20260928 06:37:58.965917 19896 heartbeater.cc:503] Master 127.5.22.62:39465 was elected leader, sending a full tablet report...
I20260928 06:37:58.966276 20043 ts_tablet_manager.cc:1428] T 12e8c8877cd6418b98a9073b780bf8b0 P ba8bcffaeb5a4a4eafe60c4a77c6f513: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260928 06:37:58.983568 19757 tablet.cc:2369] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:37:59.034670 20046 raft_consensus.cc:482] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:37:59.034930 20046 raft_consensus.cc:504] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } }
I20260928 06:37:59.036062 20046 leader_election.cc:288] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba8bcffaeb5a4a4eafe60c4a77c6f513 (127.5.22.2:32797), 8afe0ccc5aaf41e68c49622dc18d96ae (127.5.22.1:40803)
I20260928 06:37:59.039745 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:59.036290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:32797 (local address 127.5.22.3:46537)
0928 06:37:59.036404 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:59.036420 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:59.036515 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:59.036871 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:59.036874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:59.036903 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:59.037001 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:59.037006 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:59.037747 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:59.037752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:59.038625 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:59.038635 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:59.039293 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:59.039297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:59.039417 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:59.039435 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 06:37:59.039535 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 06:37:59.039611 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 06:37:59.039745 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:59.036395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:46537 (local address 127.5.22.2:32797)
0928 06:37:59.036520 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:59.036527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:59.036546 (+ 19us) server_negotiation.cc:407] Connection header received
0928 06:37:59.036603 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:59.036608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:59.036672 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:59.036759 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:59.037124 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:59.037633 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:59.038750 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:59.039166 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:59.039289 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:59.039353 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:59.039550 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 06:37:59.039620 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:37:59.039947 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:59.036416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:56215 (local address 127.5.22.1:40803)
0928 06:37:59.036832 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:59.036836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:59.036850 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:37:59.036988 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:59.036991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:59.037046 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:59.037158 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:59.037418 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:59.037915 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:59.038981 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:59.039427 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:59.039545 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:59.039713 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:59.039764 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 06:37:59.039828 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":102,"threads_started":1}
I20260928 06:37:59.039956 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:59.036290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:40803 (local address 127.5.22.3:56215)
0928 06:37:59.036759 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:59.036773 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:59.036853 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:59.037167 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:59.037170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:59.037184 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:59.037302 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:59.037308 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:59.038023 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:59.038028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:59.038854 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:59.038863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:59.039547 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:59.039551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:59.039654 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:59.039674 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 06:37:59.039732 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 06:37:59.039799 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":129,"thread_start_us":114,"threads_started":1}
I20260928 06:37:59.040457 19850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e8c8877cd6418b98a9073b780bf8b0" candidate_uuid: "0bc647d664234b94808b88315a39b22f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" is_pre_election: true
I20260928 06:37:59.040457 19710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e8c8877cd6418b98a9073b780bf8b0" candidate_uuid: "0bc647d664234b94808b88315a39b22f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" is_pre_election: true
I20260928 06:37:59.040771 19710 raft_consensus.cc:2454] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bc647d664234b94808b88315a39b22f in term 0.
I20260928 06:37:59.040787 19850 raft_consensus.cc:2454] T 12e8c8877cd6418b98a9073b780bf8b0 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bc647d664234b94808b88315a39b22f in term 0.
I20260928 06:37:59.041254 19915 leader_election.cc:302] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc647d664234b94808b88315a39b22f, 8afe0ccc5aaf41e68c49622dc18d96ae; no voters:
I20260928 06:37:59.041551 20046 raft_consensus.cc:2781] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:37:59.041627 20046 raft_consensus.cc:482] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:37:59.041692 20046 raft_consensus.cc:3037] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:59.042835 20046 raft_consensus.cc:504] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } }
I20260928 06:37:59.043190 20046 leader_election.cc:288] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f [CANDIDATE]: Term 1 election: Requested vote from peers ba8bcffaeb5a4a4eafe60c4a77c6f513 (127.5.22.2:32797), 8afe0ccc5aaf41e68c49622dc18d96ae (127.5.22.1:40803)
I20260928 06:37:59.043542 19850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e8c8877cd6418b98a9073b780bf8b0" candidate_uuid: "0bc647d664234b94808b88315a39b22f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513"
I20260928 06:37:59.043542 19710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e8c8877cd6418b98a9073b780bf8b0" candidate_uuid: "0bc647d664234b94808b88315a39b22f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae"
I20260928 06:37:59.043668 19850 raft_consensus.cc:3037] T 12e8c8877cd6418b98a9073b780bf8b0 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:59.043694 19710 raft_consensus.cc:3037] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:59.044953 19710 raft_consensus.cc:2454] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc647d664234b94808b88315a39b22f in term 1.
I20260928 06:37:59.044966 19850 raft_consensus.cc:2454] T 12e8c8877cd6418b98a9073b780bf8b0 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc647d664234b94808b88315a39b22f in term 1.
I20260928 06:37:59.045256 19917 leader_election.cc:302] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc647d664234b94808b88315a39b22f, ba8bcffaeb5a4a4eafe60c4a77c6f513; no voters:
I20260928 06:37:59.045483 20046 raft_consensus.cc:2781] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:59.045730 20046 raft_consensus.cc:686] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f [term 1 LEADER]: Becoming Leader. State: Replica: 0bc647d664234b94808b88315a39b22f, State: Running, Role: LEADER
I20260928 06:37:59.046015 20046 consensus_queue.cc:237] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } }
I20260928 06:37:59.049168 19565 catalog_manager.cc:5495] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bc647d664234b94808b88315a39b22f (127.5.22.3). New cstate: current_term: 1 leader_uuid: "0bc647d664234b94808b88315a39b22f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260928 06:37:59.063463 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:59.059696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49132 (local address 127.5.22.3:34975)
0928 06:37:59.059852 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:59.059856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:59.059871 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:37:59.060066 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:59.060069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:59.060134 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:59.060244 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:37:59.060477 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:59.061043 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:59.061898 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:59.062131 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:59.062240 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:59.062620 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:37:59.063028 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:37:59.063167 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:59.063229 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 06:37:59.063286 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 06:37:59.069412 19850 raft_consensus.cc:1261] T 12e8c8877cd6418b98a9073b780bf8b0 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [term 1 FOLLOWER]: Refusing update from remote peer 0bc647d664234b94808b88315a39b22f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:59.069464 19710 raft_consensus.cc:1261] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Refusing update from remote peer 0bc647d664234b94808b88315a39b22f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:59.070106 20046 consensus_queue.cc:1035] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:59.070312 20050 consensus_queue.cc:1035] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:59.075903 20053 mvcc.cc:205] Tried to move back new op lower bound from 7334205354260119552 to 7334205354175037440. Current Snapshot: MvccSnapshot[applied={T|T < 7334205354260119552}]
I20260928 06:37:59.085657 19566 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:36390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:37:59.085813 19566 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:36390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:37:59.088255 19566 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63: Sending DeleteTablet for 3 replicas of tablet 12e8c8877cd6418b98a9073b780bf8b0
I20260928 06:37:59.089102 19690 tablet_service.cc:1514] Processing DeleteTablet for tablet 12e8c8877cd6418b98a9073b780bf8b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:59 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:55348
I20260928 06:37:59.089138 19829 tablet_service.cc:1514] Processing DeleteTablet for tablet 12e8c8877cd6418b98a9073b780bf8b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:59 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:39172
I20260928 06:37:59.089244 19962 tablet_service.cc:1514] Processing DeleteTablet for tablet 12e8c8877cd6418b98a9073b780bf8b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:59 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:49126
I20260928 06:37:59.089694 20064 tablet_replica.cc:330] T 12e8c8877cd6418b98a9073b780bf8b0 P ba8bcffaeb5a4a4eafe60c4a77c6f513: stopping tablet replica
I20260928 06:37:59.089720 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 19604
I20260928 06:37:59.089895 20063 tablet_replica.cc:330] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae: stopping tablet replica
I20260928 06:37:59.090040 20064 raft_consensus.cc:2229] T 12e8c8877cd6418b98a9073b780bf8b0 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:37:59.090097 20063 raft_consensus.cc:2229] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:37:59.090394 20064 raft_consensus.cc:2258] T 12e8c8877cd6418b98a9073b780bf8b0 P ba8bcffaeb5a4a4eafe60c4a77c6f513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:59.090792 20065 tablet_replica.cc:330] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f: stopping tablet replica
I20260928 06:37:59.091053 20065 raft_consensus.cc:2229] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:37:59.091434 20065 raft_consensus.cc:2258] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:59.093037 20065 ts_tablet_manager.cc:1905] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:37:59.093531 20064 ts_tablet_manager.cc:1905] T 12e8c8877cd6418b98a9073b780bf8b0 P ba8bcffaeb5a4a4eafe60c4a77c6f513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:37:59.095815 20065 ts_tablet_manager.cc:1918] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:37:59.095924 20065 log.cc:1196] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/12e8c8877cd6418b98a9073b780bf8b0
I20260928 06:37:59.096271 20065 ts_tablet_manager.cc:1939] T 12e8c8877cd6418b98a9073b780bf8b0 P 0bc647d664234b94808b88315a39b22f: Deleting consensus metadata
I20260928 06:37:59.096938 20064 ts_tablet_manager.cc:1918] T 12e8c8877cd6418b98a9073b780bf8b0 P ba8bcffaeb5a4a4eafe60c4a77c6f513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:37:59.097056 20064 log.cc:1196] T 12e8c8877cd6418b98a9073b780bf8b0 P ba8bcffaeb5a4a4eafe60c4a77c6f513: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/wal/wals/12e8c8877cd6418b98a9073b780bf8b0
I20260928 06:37:59.097401 20064 ts_tablet_manager.cc:1939] T 12e8c8877cd6418b98a9073b780bf8b0 P ba8bcffaeb5a4a4eafe60c4a77c6f513: Deleting consensus metadata
I20260928 06:37:59.097457 19551 catalog_manager.cc:4841] TS 0bc647d664234b94808b88315a39b22f (127.5.22.3:34975): tablet 12e8c8877cd6418b98a9073b780bf8b0 (table test-table [id=66ae5b1261644cb5b2784e03ba40efc9]) successfully deleted
I20260928 06:37:59.098667 19552 catalog_manager.cc:4841] TS ba8bcffaeb5a4a4eafe60c4a77c6f513 (127.5.22.2:32797): tablet 12e8c8877cd6418b98a9073b780bf8b0 (table test-table [id=66ae5b1261644cb5b2784e03ba40efc9]) successfully deleted
W20260928 06:37:59.098906 19550 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.22.1:40803 (error 108)
I20260928 06:37:59.099854 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:59.099569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:40803 (local address 127.0.0.1:55352)
0928 06:37:59.099722 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:59.099781 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.1:40803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 06:37:59.100121 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 19762
W20260928 06:37:59.100011 19550 catalog_manager.cc:4584] TS 8afe0ccc5aaf41e68c49622dc18d96ae (127.5.22.1:40803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12e8c8877cd6418b98a9073b780bf8b0: Network error: Client connection negotiation failed: client connection to 127.5.22.1:40803: connect: Connection refused (error 111)
I20260928 06:37:59.107138 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 19899
I20260928 06:37:59.113000 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 19533
I20260928 06:37:59.119720 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:39465
--webserver_interface=127.5.22.62
--webserver_port=45167
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:41197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:39465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:59.231305 20067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:59.231611 20067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:59.231684 20067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:59.235741 20067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:59.235838 20067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:59.235878 20067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:59.235931 20067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:59.235975 20067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:59.241519 20067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:41197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:39465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:39465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=45167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:59.242703 20067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:59.243745 20067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:37:59.252722 20067 server_base.cc:1034] running on GCE node
W20260928 06:37:59.252653 20072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:59.252669 20073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:59.252653 20075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:59.253509 20067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:59.254690 20067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:59.255851 20067 hybrid_clock.cc:648] HybridClock initialized: now 1790577479255835 us; error 29 us; skew 500 ppm
Sep 28 06:37:59 dist-test-slave-j0qf krb5kdc[18980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:59.258659 20067 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:37:59.259549 20067 webserver.cc:460] Webserver started at http://127.5.22.62:45167/ using document root <none> and password file <none>
I20260928 06:37:59.260129 20067 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:59.260192 20067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:59.263983 20067 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 06:37:59.266022 20082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:59.267298 20067 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:37:59.267467 20067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "34e41488fd0749dd8517addf86daae63"
format_stamp: "Formatted at 2026-09-28 06:37:57 on dist-test-slave-j0qf"
server_key: "8f751842537442d359b56869617f1c54"
server_key_iv: "f6219250bb9897a38249920182d79756"
server_key_version: "encryptionkey@0"
I20260928 06:37:59.268038 20067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:59.286134 20067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:59.288982 20067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:59.289206 20067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:59.296473 20067 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:39465
I20260928 06:37:59.296726 20135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:39465 every 8 connection(s)
I20260928 06:37:59.297593 20067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:59.301831 20136 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:37:59.303483 20136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63: Bootstrap starting.
I20260928 06:37:59.306605 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 20067
I20260928 06:37:59.307027 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:40803
--local_ip_for_outbound_sockets=127.5.22.1
--tserver_master_addrs=127.5.22.62:39465
--webserver_port=45035
--webserver_interface=127.5.22.1
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:41197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:37:59.326913 20136 log.cc:824] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:59.341627 20136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:37:59.342087 20136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63: Bootstrap complete.
I20260928 06:37:59.347508 20136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e41488fd0749dd8517addf86daae63" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 39465 } }
I20260928 06:37:59.347998 20136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e41488fd0749dd8517addf86daae63, State: Initialized, Role: FOLLOWER
I20260928 06:37:59.348541 20136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e41488fd0749dd8517addf86daae63" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 39465 } }
I20260928 06:37:59.348683 20136 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:59.348752 20136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:59.348847 20136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [term 2 FOLLOWER]: Advancing to term 3
I20260928 06:37:59.351236 20136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e41488fd0749dd8517addf86daae63" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 39465 } }
I20260928 06:37:59.351576 20136 leader_election.cc:302] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34e41488fd0749dd8517addf86daae63; no voters:
I20260928 06:37:59.351900 20136 leader_election.cc:288] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 06:37:59.352028 20140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 06:37:59.352298 20140 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [term 3 LEADER]: Becoming Leader. State: Replica: 34e41488fd0749dd8517addf86daae63, State: Running, Role: LEADER
I20260928 06:37:59.352629 20140 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e41488fd0749dd8517addf86daae63" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 39465 } }
I20260928 06:37:59.353176 20136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:59.353682 20142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34e41488fd0749dd8517addf86daae63. Latest consensus state: current_term: 3 leader_uuid: "34e41488fd0749dd8517addf86daae63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e41488fd0749dd8517addf86daae63" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 39465 } } }
I20260928 06:37:59.353816 20142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:59.354012 20141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "34e41488fd0749dd8517addf86daae63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e41488fd0749dd8517addf86daae63" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 39465 } } }
I20260928 06:37:59.354116 20141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34e41488fd0749dd8517addf86daae63 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:59.354183 20144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:59.359375 20144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:59.360002 20144 catalog_manager.cc:1259] Loaded cluster ID: 6c0d22fc78814b4da244d4b25fd051f8
I20260928 06:37:59.360069 20144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 06:37:59.361061 20144 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f36ea8389ad google::LogMessage::Fail() at ??:0
@ 0x7f36ea83b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f36ea8384de google::LogMessage::Flush() at ??:0
@ 0x7f36ea839a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f36f5177fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f36f5178d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f36f5175579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f36f51b9640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f36eca8a3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f36eca8ac83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f36eca8c3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f36eca7b512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f36ee0bf6db start_thread at ??:0
@ 0x7f36e933661f clone at ??:0
W20260928 06:37:59.457702 20138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:59.457989 20138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:59.458065 20138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:59.462118 20138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:59.462206 20138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:59.462297 20138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:37:59.467787 20138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:41197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:40803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=45035
--tserver_master_addrs=127.5.22.62:39465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:59.468930 20138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:59.469985 20138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:59.479991 20177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:59.480018 20180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:59.479998 20178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:59.480353 20138 server_base.cc:1034] running on GCE node
I20260928 06:37:59.480752 20138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:59.481423 20138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:59.482610 20138 hybrid_clock.cc:648] HybridClock initialized: now 1790577479482581 us; error 52 us; skew 500 ppm
Sep 28 06:37:59 dist-test-slave-j0qf krb5kdc[18980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:59.485584 20138 init.cc:377] Logged in from keytab as kudu/127.5.22.1@KRBTEST.COM (short username kudu)
I20260928 06:37:59.486559 20138 webserver.cc:460] Webserver started at http://127.5.22.1:45035/ using document root <none> and password file <none>
I20260928 06:37:59.487124 20138 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:59.487217 20138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:59.491171 20138 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 06:37:59.493418 20187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:59.494807 20138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:37:59.494964 20138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "8afe0ccc5aaf41e68c49622dc18d96ae"
format_stamp: "Formatted at 2026-09-28 06:37:57 on dist-test-slave-j0qf"
server_key: "6189a654d4afaa2a89fc2da02c880f2c"
server_key_iv: "393a0a9c61976d80018652eb2fa3e9ad"
server_key_version: "encryptionkey@0"
I20260928 06:37:59.495520 20138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:59.508093 20138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:59.511224 20138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:59.511432 20138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:59.512027 20138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:59.513156 20194 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:37:59.518509 20138 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:37:59.518585 20138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260928 06:37:59.518669 20138 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:37:59.520332 20138 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:37:59.520426 20138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 06:37:59.520709 20194 tablet_bootstrap.cc:492] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae: Bootstrap starting.
I20260928 06:37:59.531729 20138 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:40803
I20260928 06:37:59.532819 20138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:37:59.533960 20302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:40803 every 8 connection(s)
I20260928 06:37:59.535245 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 20138
I20260928 06:37:59.535858 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:32797
--local_ip_for_outbound_sockets=127.5.22.2
--tserver_master_addrs=127.5.22.62:39465
--webserver_port=40281
--webserver_interface=127.5.22.2
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:41197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:37:59.541640 20194 log.cc:824] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:59.548430 20194 tablet_bootstrap.cc:492] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:37:59.548801 20194 tablet_bootstrap.cc:492] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae: Bootstrap complete.
I20260928 06:37:59.549927 20194 ts_tablet_manager.cc:1397] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae: Time spent bootstrapping tablet: real 0.029s user 0.022s sys 0.004s
I20260928 06:37:59.552476 20194 raft_consensus.cc:348] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } }
I20260928 06:37:59.552922 20194 raft_consensus.cc:729] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8afe0ccc5aaf41e68c49622dc18d96ae, State: Initialized, Role: FOLLOWER
I20260928 06:37:59.553417 20194 consensus_queue.cc:260] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } }
I20260928 06:37:59.554543 20194 ts_tablet_manager.cc:1428] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260928 06:37:59.612452 20305 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:59.535206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.1:59663)
0928 06:37:59.535764 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:59.535820 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:59.536855 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:59.611460 (+ 74605us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":180,"threads_started":1}
W20260928 06:37:59.613073 20303 heartbeater.cc:644] Failed to heartbeat to 127.5.22.62:39465 (0 consecutive failures): Network error: Failed to ping master at 127.5.22.62:39465: Client connection negotiation failed: client connection to 127.5.22.62:39465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:37:59.613848 20305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:59.613558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.1:55375)
0928 06:37:59.613703 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:59.613783 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 06:37:59.614533 20305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:59.614285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.1:39933)
0928 06:37:59.614421 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:59.614477 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 06:37:59.614719 20303 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:37:59.672360 20306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:59.672614 20306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:59.672657 20306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:59.676769 20306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:59.676827 20306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:59.676923 20306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:37:59.682221 20306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:41197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:32797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=40281
--tserver_master_addrs=127.5.22.62:39465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:59.683334 20306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:59.684348 20306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:59.694187 20316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:59.694188 20313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:59.694453 20306 server_base.cc:1034] running on GCE node
W20260928 06:37:59.694188 20314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:59.694983 20306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:59.695601 20306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:59.696794 20306 hybrid_clock.cc:648] HybridClock initialized: now 1790577479696755 us; error 51 us; skew 500 ppm
Sep 28 06:37:59 dist-test-slave-j0qf krb5kdc[18980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:59.699680 20306 init.cc:377] Logged in from keytab as kudu/127.5.22.2@KRBTEST.COM (short username kudu)
I20260928 06:37:59.700551 20306 webserver.cc:460] Webserver started at http://127.5.22.2:40281/ using document root <none> and password file <none>
I20260928 06:37:59.701077 20306 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:59.701136 20306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:59.705042 20306 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:37:59.707083 20323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:59.708273 20306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:37:59.708369 20306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513"
format_stamp: "Formatted at 2026-09-28 06:37:57 on dist-test-slave-j0qf"
server_key: "dd1a1d571a7462c99551d426c968baac"
server_key_iv: "08f6c7a64c0290f9de31ce4774d8f82c"
server_key_version: "encryptionkey@0"
I20260928 06:37:59.708894 20306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:59.729452 20306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:59.732960 20306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:59.733122 20306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:59.733681 20306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:59.734519 20306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:59.734567 20306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:59.734609 20306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:59.734624 20306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:59.744023 20306 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:32797
I20260928 06:37:59.744148 20436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:32797 every 8 connection(s)
I20260928 06:37:59.745113 20306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:37:59.749351 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:59.747500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.2:36773)
0928 06:37:59.748002 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:59.748135 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":120,"threads_started":1}
W20260928 06:37:59.749907 20437 heartbeater.cc:644] Failed to heartbeat to 127.5.22.62:39465 (0 consecutive failures): Network error: Failed to ping master at 127.5.22.62:39465: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
I20260928 06:37:59.750646 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:59.750405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.2:42719)
0928 06:37:59.750535 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:59.750589 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 06:37:59.751319 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:59.751124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.2:33475)
0928 06:37:59.751226 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:59.751269 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 06:37:59.751537 20437 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:37:59.754208 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 20306
I20260928 06:37:59.754631 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:34975
--local_ip_for_outbound_sockets=127.5.22.3
--tserver_master_addrs=127.5.22.62:39465
--webserver_port=42499
--webserver_interface=127.5.22.3
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:41197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:59.870000 20440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:59.870342 20440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:59.870390 20440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:59.874665 20440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:59.874761 20440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:59.874866 20440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:37:59.880383 20440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:41197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:34975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=42499
--tserver_master_addrs=127.5.22.62:39465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:37:59.881480 20440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:59.882596 20440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:59.893035 20446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:59.893013 20445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:59.893136 20449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:59.893517 20440 server_base.cc:1034] running on GCE node
I20260928 06:37:59.893914 20440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:59.894599 20440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:59.895808 20440 hybrid_clock.cc:648] HybridClock initialized: now 1790577479895769 us; error 51 us; skew 500 ppm
Sep 28 06:37:59 dist-test-slave-j0qf krb5kdc[18980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:59.899062 20440 init.cc:377] Logged in from keytab as kudu/127.5.22.3@KRBTEST.COM (short username kudu)
I20260928 06:37:59.899940 20440 webserver.cc:460] Webserver started at http://127.5.22.3:42499/ using document root <none> and password file <none>
I20260928 06:37:59.900507 20440 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:59.900574 20440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:59.904922 20440 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:37:59.907271 20456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:59.908682 20440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:37:59.908849 20440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "0bc647d664234b94808b88315a39b22f"
format_stamp: "Formatted at 2026-09-28 06:37:57 on dist-test-slave-j0qf"
server_key: "40911fae3b63f3fe72cd46c8e1047756"
server_key_iv: "e73d379fcd449c1e54699268b6697522"
server_key_version: "encryptionkey@0"
I20260928 06:37:59.909446 20440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:59.929211 20440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:59.932723 20440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:59.932936 20440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:59.933555 20440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:59.934397 20440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:59.934468 20440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:59.934537 20440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:59.934580 20440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:59.943753 20440 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:34975
I20260928 06:37:59.943786 20571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:34975 every 8 connection(s)
I20260928 06:37:59.944744 20440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:37:59.948648 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:59.946906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.3:51637)
0928 06:37:59.947394 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:59.947515 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":129,"threads_started":1}
W20260928 06:37:59.949096 20572 heartbeater.cc:644] Failed to heartbeat to 127.5.22.62:39465 (0 consecutive failures): Network error: Failed to ping master at 127.5.22.62:39465: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
I20260928 06:37:59.949776 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:59.949550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.3:37005)
0928 06:37:59.949665 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:59.949714 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 06:37:59.950407 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:59.950176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.3:50909)
0928 06:37:59.950279 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:59.950346 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 06:37:59.950587 20572 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:37:59.951486 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 20440
I20260928 06:38:00.615856 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:00.615373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.1:44043)
0928 06:38:00.615670 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:00.615739 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":85,"threads_started":1}
I20260928 06:38:00.752521 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:00.752086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.2:51461)
0928 06:38:00.752331 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:00.752381 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":79,"threads_started":1}
I20260928 06:38:00.951524 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:00.951104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.3:49491)
0928 06:38:00.951356 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:00.951409 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":72,"threads_started":1}
I20260928 06:38:01.354684 20579 raft_consensus.cc:482] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:38:01.354987 20579 raft_consensus.cc:504] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } }
I20260928 06:38:01.356138 20579 leader_election.cc:288] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0bc647d664234b94808b88315a39b22f (127.5.22.3:34975), ba8bcffaeb5a4a4eafe60c4a77c6f513 (127.5.22.2:32797)
Sep 28 06:38:01 dist-test-slave-j0qf krb5kdc[18980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for kudu/127.5.22.3@KRBTEST.COM
Sep 28 06:38:01 dist-test-slave-j0qf krb5kdc[18980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for kudu/127.5.22.2@KRBTEST.COM
I20260928 06:38:01.370496 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:01.356460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:32797 (local address 127.5.22.1:59367)
0928 06:38:01.356808 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:01.356824 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:01.356961 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:01.358882 (+ 1921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:01.358888 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:01.359421 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:01.359895 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:01.359910 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:01.361216 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:01.361219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:01.362100 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:01.362109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:01.362904 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:01.362908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:01.363100 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:01.363805 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:01.363817 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:01.365712 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:01.367924 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:01.367928 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:01.367929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:01.368284 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:01.368753 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:01.368755 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:01.368757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:01.368903 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:01.369682 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:01.369687 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:01.369899 (+ 212us) client_negotiation.cc:769] Sending connection context
0928 06:38:01.370048 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 06:38:01.370259 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":19,"spinlock_wait_cycles":3456,"thread_start_us":83,"threads_started":1}
I20260928 06:38:01.370631 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:01.356630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:59367 (local address 127.5.22.2:32797)
0928 06:38:01.357080 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:01.357097 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:01.357462 (+ 365us) server_negotiation.cc:407] Connection header received
0928 06:38:01.358101 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:01.358107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:01.358514 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:01.358817 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:01.360070 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:01.361061 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:01.362225 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:01.362758 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:01.362946 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:01.365894 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:01.365898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:01.365907 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:01.365934 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:01.367803 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:01.368484 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:01.368491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:01.368498 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:01.368654 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:01.368990 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:01.368992 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:01.368993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:01.369553 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:01.369683 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:01.370161 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 06:38:01.370455 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":96,"threads_started":1}
I20260928 06:38:01.370930 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:01.356480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:34975 (local address 127.5.22.1:36711)
0928 06:38:01.357065 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:01.357080 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:01.357188 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:01.358882 (+ 1694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:01.358888 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:01.359419 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:01.359895 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:01.359912 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:01.361209 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:01.361219 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:01.362103 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:01.362112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:01.362879 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:01.362883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:01.363104 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:01.363740 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:01.363756 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:01.365335 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:01.367778 (+ 2443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:01.367784 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:01.367804 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:01.368412 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:01.368993 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:01.368995 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:01.368996 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:01.369065 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:01.370413 (+ 1348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:01.370416 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:01.370508 (+ 92us) client_negotiation.cc:769] Sending connection context
0928 06:38:01.370616 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 06:38:01.370765 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":154,"spinlock_wait_cycles":3712,"thread_start_us":66,"threads_started":1}
I20260928 06:38:01.371585 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:01.356630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:36711 (local address 127.5.22.3:34975)
0928 06:38:01.357259 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:01.357279 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:01.357319 (+ 40us) server_negotiation.cc:407] Connection header received
0928 06:38:01.358090 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:01.358096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:01.358513 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:01.358816 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:01.360093 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:01.361060 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:01.362231 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:01.362752 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:01.362929 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:01.365603 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:01.365610 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:01.365629 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:01.365665 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:01.367651 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:01.368522 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:01.368527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:01.368531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:01.368653 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:01.369150 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:01.369153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:01.369155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:01.369563 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:01.369683 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:01.371231 (+ 1548us) server_negotiation.cc:299] Negotiation successful
0928 06:38:01.371448 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":125,"threads_started":1}
W20260928 06:38:01.373682 20189 leader_election.cc:341] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0bc647d664234b94808b88315a39b22f (127.5.22.3:34975): Not found: Tablet not found: 12e8c8877cd6418b98a9073b780bf8b0
W20260928 06:38:01.373864 20190 leader_election.cc:341] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ba8bcffaeb5a4a4eafe60c4a77c6f513 (127.5.22.2:32797): Not found: Tablet not found: 12e8c8877cd6418b98a9073b780bf8b0
I20260928 06:38:01.373975 20190 leader_election.cc:302] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8afe0ccc5aaf41e68c49622dc18d96ae; no voters: 0bc647d664234b94808b88315a39b22f, ba8bcffaeb5a4a4eafe60c4a77c6f513
I20260928 06:38:01.374432 20579 raft_consensus.cc:2726] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:38:01.617079 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:01.616736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.1:50661)
0928 06:38:01.616912 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:01.616979 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 06:38:01.753734 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:01.753281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.2:50785)
0928 06:38:01.753557 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:01.753616 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
I20260928 06:38:01.952744 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:01.952337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.3:49283)
0928 06:38:01.952592 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:01.952649 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":80,"threads_started":1}
I20260928 06:38:02.618397 20586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:02.617870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.1:37507)
0928 06:38:02.618167 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:02.618230 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":78,"threads_started":1}
I20260928 06:38:02.755213 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:02.754774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.2:58629)
0928 06:38:02.755051 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:02.755115 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":81,"threads_started":1}
I20260928 06:38:02.954134 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:02.953637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.3:39251)
0928 06:38:02.953955 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:02.954018 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":83,"threads_started":1}
I20260928 06:38:03.276206 20589 raft_consensus.cc:482] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:38:03.276460 20589 raft_consensus.cc:504] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } }
I20260928 06:38:03.276945 20589 leader_election.cc:288] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0bc647d664234b94808b88315a39b22f (127.5.22.3:34975), ba8bcffaeb5a4a4eafe60c4a77c6f513 (127.5.22.2:32797)
W20260928 06:38:03.277839 20190 leader_election.cc:341] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ba8bcffaeb5a4a4eafe60c4a77c6f513 (127.5.22.2:32797): Not found: Tablet not found: 12e8c8877cd6418b98a9073b780bf8b0
W20260928 06:38:03.277951 20189 leader_election.cc:341] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0bc647d664234b94808b88315a39b22f (127.5.22.3:34975): Not found: Tablet not found: 12e8c8877cd6418b98a9073b780bf8b0
I20260928 06:38:03.278002 20189 leader_election.cc:302] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8afe0ccc5aaf41e68c49622dc18d96ae; no voters: 0bc647d664234b94808b88315a39b22f, ba8bcffaeb5a4a4eafe60c4a77c6f513
I20260928 06:38:03.278157 20589 raft_consensus.cc:2726] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:38:03.619681 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:03.619241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.1:49183)
0928 06:38:03.619516 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:03.619578 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
I20260928 06:38:03.756604 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:03.756123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.2:43415)
0928 06:38:03.756444 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:03.756511 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":77,"threads_started":1}
I20260928 06:38:03.955611 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:03.955097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.3:41353)
0928 06:38:03.955443 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:03.955516 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":92,"threads_started":1}
I20260928 06:38:04.621168 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.620694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.1:35613)
0928 06:38:04.621009 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.621070 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":79,"threads_started":1}
I20260928 06:38:04.757920 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.757450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.2:42019)
0928 06:38:04.757732 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.757791 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":111,"threads_started":1}
I20260928 06:38:04.956964 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.956564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.3:60827)
0928 06:38:04.956812 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.956865 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":85,"threads_started":1}
I20260928 06:38:05.209594 20596 raft_consensus.cc:482] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:38:05.209723 20596 raft_consensus.cc:504] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } }
I20260928 06:38:05.210201 20596 leader_election.cc:288] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0bc647d664234b94808b88315a39b22f (127.5.22.3:34975), ba8bcffaeb5a4a4eafe60c4a77c6f513 (127.5.22.2:32797)
W20260928 06:38:05.211050 20190 leader_election.cc:341] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ba8bcffaeb5a4a4eafe60c4a77c6f513 (127.5.22.2:32797): Not found: Tablet not found: 12e8c8877cd6418b98a9073b780bf8b0
W20260928 06:38:05.211145 20189 leader_election.cc:341] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0bc647d664234b94808b88315a39b22f (127.5.22.3:34975): Not found: Tablet not found: 12e8c8877cd6418b98a9073b780bf8b0
I20260928 06:38:05.211257 20189 leader_election.cc:302] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8afe0ccc5aaf41e68c49622dc18d96ae; no voters: 0bc647d664234b94808b88315a39b22f, ba8bcffaeb5a4a4eafe60c4a77c6f513
I20260928 06:38:05.211422 20596 raft_consensus.cc:2726] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:38:05.622524 20597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:05.621956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.1:33807)
0928 06:38:05.622269 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:05.622401 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":101,"threads_started":1}
I20260928 06:38:05.759146 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:05.758746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.2:60263)
0928 06:38:05.758998 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:05.759054 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":87,"threads_started":1}
I20260928 06:38:05.958698 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:05.957834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.3:33547)
0928 06:38:05.958171 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:05.958237 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":104,"threads_started":1}
I20260928 06:38:06.624058 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:06.623530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.1:46671)
0928 06:38:06.623843 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:06.623926 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":101,"threads_started":1}
I20260928 06:38:06.760478 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:06.759956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.2:42611)
0928 06:38:06.760287 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:06.760367 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":77,"threads_started":1}
I20260928 06:38:06.960063 20602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:06.959564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.3:33177)
0928 06:38:06.959900 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:06.959968 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":89,"threads_started":1}
I20260928 06:38:07.024797 20603 raft_consensus.cc:482] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:38:07.024963 20603 raft_consensus.cc:504] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } }
I20260928 06:38:07.025506 20603 leader_election.cc:288] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0bc647d664234b94808b88315a39b22f (127.5.22.3:34975), ba8bcffaeb5a4a4eafe60c4a77c6f513 (127.5.22.2:32797)
W20260928 06:38:07.026258 20189 leader_election.cc:341] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0bc647d664234b94808b88315a39b22f (127.5.22.3:34975): Not found: Tablet not found: 12e8c8877cd6418b98a9073b780bf8b0
W20260928 06:38:07.026425 20190 leader_election.cc:341] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ba8bcffaeb5a4a4eafe60c4a77c6f513 (127.5.22.2:32797): Not found: Tablet not found: 12e8c8877cd6418b98a9073b780bf8b0
I20260928 06:38:07.026505 20190 leader_election.cc:302] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8afe0ccc5aaf41e68c49622dc18d96ae; no voters: 0bc647d664234b94808b88315a39b22f, ba8bcffaeb5a4a4eafe60c4a77c6f513
I20260928 06:38:07.026666 20603 raft_consensus.cc:2726] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:38:07.625252 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:07.624843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.1:33485)
0928 06:38:07.625098 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:07.625157 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":77,"threads_started":1}
I20260928 06:38:07.761720 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:07.761275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.2:38075)
0928 06:38:07.761562 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:07.761624 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":104,"threads_started":1}
I20260928 06:38:07.961335 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:07.960918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.3:58287)
0928 06:38:07.961183 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:07.961237 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":89,"threads_started":1}
I20260928 06:38:08.626734 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:08.626191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.1:36501)
0928 06:38:08.626551 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:08.626632 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":84,"threads_started":1}
I20260928 06:38:08.762984 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:08.762531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.2:48719)
0928 06:38:08.762822 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:08.762882 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":79,"threads_started":1}
I20260928 06:38:08.962683 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:08.962170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.3:53119)
0928 06:38:08.962519 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:08.962585 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":79,"threads_started":1}
I20260928 06:38:09.628108 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.627651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.1:40355)
0928 06:38:09.627956 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:09.628015 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":97,"threads_started":1}
I20260928 06:38:09.764475 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.764020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.2:46913)
0928 06:38:09.764311 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:09.764370 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":98,"threads_started":1}
I20260928 06:38:09.963893 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.963497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.3:46931)
0928 06:38:09.963744 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:09.963800 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":77,"threads_started":1}
I20260928 06:38:10.629401 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:10.628938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.1:36905)
0928 06:38:10.629238 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:10.629302 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":91,"threads_started":1}
I20260928 06:38:10.765612 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:10.765220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.2:44417)
0928 06:38:10.765465 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:10.765519 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260928 06:38:10.965361 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:10.964801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.3:47197)
0928 06:38:10.965130 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:10.965193 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":124,"threads_started":1}
I20260928 06:38:11.630842 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:11.630295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.1:60713)
0928 06:38:11.630668 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:11.630736 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":88,"threads_started":1}
I20260928 06:38:11.766739 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:11.766274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.2:49411)
0928 06:38:11.766589 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:11.766645 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":143,"threads_started":1}
I20260928 06:38:11.966709 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:11.966206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.3:37965)
0928 06:38:11.966509 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:11.966571 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":132,"threads_started":1}
I20260928 06:38:12.632120 20619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:12.631700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.1:40899)
0928 06:38:12.631963 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:12.632023 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":77,"threads_started":1}
I20260928 06:38:12.768209 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:12.767738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.2:48901)
0928 06:38:12.768015 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:12.768096 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":97,"threads_started":1}
I20260928 06:38:12.968266 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:12.967696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.3:43785)
0928 06:38:12.968099 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:12.968166 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":142,"threads_started":1}
I20260928 06:38:13.233747 20623 raft_consensus.cc:482] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:38:13.233923 20623 raft_consensus.cc:504] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc647d664234b94808b88315a39b22f" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 34975 } } peers { permanent_uuid: "ba8bcffaeb5a4a4eafe60c4a77c6f513" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 32797 } } peers { permanent_uuid: "8afe0ccc5aaf41e68c49622dc18d96ae" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 40803 } }
I20260928 06:38:13.234499 20623 leader_election.cc:288] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0bc647d664234b94808b88315a39b22f (127.5.22.3:34975), ba8bcffaeb5a4a4eafe60c4a77c6f513 (127.5.22.2:32797)
W20260928 06:38:13.235258 20189 leader_election.cc:341] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0bc647d664234b94808b88315a39b22f (127.5.22.3:34975): Not found: Tablet not found: 12e8c8877cd6418b98a9073b780bf8b0
W20260928 06:38:13.235396 20190 leader_election.cc:341] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ba8bcffaeb5a4a4eafe60c4a77c6f513 (127.5.22.2:32797): Not found: Tablet not found: 12e8c8877cd6418b98a9073b780bf8b0
I20260928 06:38:13.235494 20190 leader_election.cc:302] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8afe0ccc5aaf41e68c49622dc18d96ae; no voters: 0bc647d664234b94808b88315a39b22f, ba8bcffaeb5a4a4eafe60c4a77c6f513
I20260928 06:38:13.235662 20623 raft_consensus.cc:2726] T 12e8c8877cd6418b98a9073b780bf8b0 P 8afe0ccc5aaf41e68c49622dc18d96ae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:38:13.633378 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:13.632948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.1:39561)
0928 06:38:13.633219 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:13.633279 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":89,"threads_started":1}
I20260928 06:38:13.769474 20625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:13.769018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.2:41927)
0928 06:38:13.769320 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:13.769378 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":94,"threads_started":1}
I20260928 06:38:13.969584 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:13.969110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.3:52533)
0928 06:38:13.969425 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:13.969487 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":93,"threads_started":1}
I20260928 06:38:14.634819 20627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:14.634389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.1:45739)
0928 06:38:14.634665 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:14.634723 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":77,"threads_started":1}
I20260928 06:38:14.770934 20628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:14.770474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.2:54333)
0928 06:38:14.770752 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:14.770811 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":74,"threads_started":1}
I20260928 06:38:14.960358 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 20138
I20260928 06:38:14.966543 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 20306
I20260928 06:38:14.971220 20629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:14.970685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:39465 (local address 127.5.22.3:52159)
0928 06:38:14.971031 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:14.971096 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":75,"threads_started":1}
I20260928 06:38:14.974567 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 20440
2026-09-28T06:38:14Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19191 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:38:15 dist-test-slave-j0qf krb5kdc[20632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:38:15 dist-test-slave-j0qf krb5kdc[20632](info): set up 2 sockets
Sep 28 06:38:15 dist-test-slave-j0qf krb5kdc[20632](info): commencing operation
krb5kdc: starting...
W20260928 06:38:16.057015 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:38:16 dist-test-slave-j0qf krb5kdc[20632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:38:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:38:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:38:16.184888 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:43541
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:43541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:38:16.298077 20645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:16.298502 20645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:16.298616 20645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:16.302784 20645 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 06:38:16.302861 20645 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 06:38:16.302893 20645 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 06:38:16.302925 20645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:38:16.302965 20645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:16.303009 20645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:16.303045 20645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:38:16.303081 20645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:38:16.308565 20645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:43541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:43541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:38:16.309703 20645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:16.310834 20645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:38:16.319710 20645 server_base.cc:1034] running on GCE node
W20260928 06:38:16.319664 20653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:16.319655 20651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:16.319665 20650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:16.320394 20645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:16.321481 20645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:16.322638 20645 hybrid_clock.cc:648] HybridClock initialized: now 1790577496322610 us; error 42 us; skew 500 ppm
Sep 28 06:38:16 dist-test-slave-j0qf krb5kdc[20632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577496, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:16.325408 20645 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:38:16.326241 20645 webserver.cc:460] Webserver started at http://127.5.22.62:41199/ using document root <none> and password file <none>
I20260928 06:38:16.326835 20645 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:16.326910 20645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:16.327073 20645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:38:16.329049 20645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "a58cb30fef744b048351d0c17588c194"
format_stamp: "Formatted at 2026-09-28 06:38:16 on dist-test-slave-j0qf"
server_key: "921bad6934aa05c44df053c02f840b14"
server_key_iv: "afb617f63704388a57903759b7fea2ea"
server_key_version: "encryptionkey@0"
I20260928 06:38:16.329552 20645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "a58cb30fef744b048351d0c17588c194"
format_stamp: "Formatted at 2026-09-28 06:38:16 on dist-test-slave-j0qf"
server_key: "921bad6934aa05c44df053c02f840b14"
server_key_iv: "afb617f63704388a57903759b7fea2ea"
server_key_version: "encryptionkey@0"
I20260928 06:38:16.333235 20645 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 06:38:16.335624 20660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:16.336833 20645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:38:16.336987 20645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "a58cb30fef744b048351d0c17588c194"
format_stamp: "Formatted at 2026-09-28 06:38:16 on dist-test-slave-j0qf"
server_key: "921bad6934aa05c44df053c02f840b14"
server_key_iv: "afb617f63704388a57903759b7fea2ea"
server_key_version: "encryptionkey@0"
I20260928 06:38:16.337088 20645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:16.354880 20645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:16.357770 20645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:16.357985 20645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:16.365346 20645 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:43541
I20260928 06:38:16.365648 20712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:43541 every 8 connection(s)
I20260928 06:38:16.366467 20645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:38:16.369174 20713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:16.371220 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 20645
I20260928 06:38:16.371354 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:38:16.371675 5208 external_mini_cluster.cc:1428] Setting key b83187431e802fee67da79ea05ae213e
I20260928 06:38:16.375805 20713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194: Bootstrap starting.
I20260928 06:38:16.377995 20713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:16.378784 20713 log.cc:824] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194: Log is configured to *not* fsync() on all Append() calls
Sep 28 06:38:16 dist-test-slave-j0qf krb5kdc[20632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:38:16.380724 20713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194: No bootstrap required, opened a new log
I20260928 06:38:16.384442 20713 raft_consensus.cc:348] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58cb30fef744b048351d0c17588c194" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43541 } }
I20260928 06:38:16.384693 20713 raft_consensus.cc:374] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:16.384801 20713 raft_consensus.cc:729] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a58cb30fef744b048351d0c17588c194, State: Initialized, Role: FOLLOWER
I20260928 06:38:16.385309 20713 consensus_queue.cc:260] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58cb30fef744b048351d0c17588c194" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43541 } }
I20260928 06:38:16.385465 20713 raft_consensus.cc:388] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:38:16.385553 20713 raft_consensus.cc:482] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:38:16.385654 20713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:16.385849 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:16.373148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58814 (local address 127.5.22.62:43541)
0928 06:38:16.373591 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:16.373606 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:16.373642 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:38:16.374282 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:16.374299 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:16.374708 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:16.375053 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:16.375887 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:16.376875 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:16.377747 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:16.378276 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:16.378499 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:16.381008 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:16.381013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:16.381025 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:16.381053 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:16.382835 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:16.383312 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:16.383317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:16.383322 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:16.383439 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:16.383745 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:16.383749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:16.383751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:16.384149 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:16.384367 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:16.384668 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 06:38:16.385004 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":141,"threads_started":1}
I20260928 06:38:16.386819 20713 raft_consensus.cc:504] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58cb30fef744b048351d0c17588c194" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43541 } }
I20260928 06:38:16.387164 20713 leader_election.cc:302] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a58cb30fef744b048351d0c17588c194; no voters:
I20260928 06:38:16.387444 20713 leader_election.cc:288] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:38:16.387521 20718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:38:16.387688 20718 raft_consensus.cc:686] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [term 1 LEADER]: Becoming Leader. State: Replica: a58cb30fef744b048351d0c17588c194, State: Running, Role: LEADER
I20260928 06:38:16.388463 20718 consensus_queue.cc:237] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58cb30fef744b048351d0c17588c194" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43541 } }
I20260928 06:38:16.388712 20713 sys_catalog.cc:564] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:38:16.390264 20720 sys_catalog.cc:455] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a58cb30fef744b048351d0c17588c194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58cb30fef744b048351d0c17588c194" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43541 } } }
I20260928 06:38:16.390420 20720 sys_catalog.cc:458] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [sys.catalog]: This master's current role is: LEADER
I20260928 06:38:16.390493 20719 sys_catalog.cc:455] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a58cb30fef744b048351d0c17588c194. Latest consensus state: current_term: 1 leader_uuid: "a58cb30fef744b048351d0c17588c194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58cb30fef744b048351d0c17588c194" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43541 } } }
I20260928 06:38:16.390587 20719 sys_catalog.cc:458] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [sys.catalog]: This master's current role is: LEADER
I20260928 06:38:16.390808 20727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:38:16.394003 20727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:38:16.399528 20727 catalog_manager.cc:1347] Generated new cluster ID: a93ffe9a51214f6b97f899075cd011ea
I20260928 06:38:16.399600 20727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:38:16.435557 20727 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:38:16.436535 20727 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:38:16.443776 20727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194: Generated new TSK 0
I20260928 06:38:16.444478 20727 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.22.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
I20260928 06:38:16.485929 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:43541
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:38:16.617585 20739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:16.617859 20739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:16.617933 20739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:16.621829 20739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:16.621918 20739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:16.622001 20739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:38:16.627136 20739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:43541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:38:16.628165 20739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:16.629166 20739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:16.638727 20747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:16.638734 20745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:16.638859 20744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:16.639047 20739 server_base.cc:1034] running on GCE node
I20260928 06:38:16.639536 20739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:16.640254 20739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:16.641438 20739 hybrid_clock.cc:648] HybridClock initialized: now 1790577496641403 us; error 54 us; skew 500 ppm
Sep 28 06:38:16 dist-test-slave-j0qf krb5kdc[20632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577496, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:16.644285 20739 init.cc:377] Logged in from keytab as kudu/127.5.22.1@KRBTEST.COM (short username kudu)
I20260928 06:38:16.645138 20739 webserver.cc:460] Webserver started at http://127.5.22.1:40357/ using document root <none> and password file <none>
I20260928 06:38:16.645704 20739 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:16.645798 20739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:16.646059 20739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:38:16.648144 20739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "76f779950ca746809a2f124299de1c55"
format_stamp: "Formatted at 2026-09-28 06:38:16 on dist-test-slave-j0qf"
server_key: "fab94183b6424fa54221b652212d737c"
server_key_iv: "1287890ecc8c241a725790426808791a"
server_key_version: "encryptionkey@0"
I20260928 06:38:16.648743 20739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "76f779950ca746809a2f124299de1c55"
format_stamp: "Formatted at 2026-09-28 06:38:16 on dist-test-slave-j0qf"
server_key: "fab94183b6424fa54221b652212d737c"
server_key_iv: "1287890ecc8c241a725790426808791a"
server_key_version: "encryptionkey@0"
I20260928 06:38:16.652562 20739 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:38:16.655073 20754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:16.656253 20739 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:38:16.656389 20739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "76f779950ca746809a2f124299de1c55"
format_stamp: "Formatted at 2026-09-28 06:38:16 on dist-test-slave-j0qf"
server_key: "fab94183b6424fa54221b652212d737c"
server_key_iv: "1287890ecc8c241a725790426808791a"
server_key_version: "encryptionkey@0"
I20260928 06:38:16.656500 20739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:16.678007 20739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:16.681416 20739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:16.681622 20739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:16.682226 20739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:16.683166 20739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:38:16.683233 20739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:16.683300 20739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:38:16.683353 20739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:16.692996 20739 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:34999
I20260928 06:38:16.693024 20868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:34999 every 8 connection(s)
I20260928 06:38:16.693913 20739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:38:16 dist-test-slave-j0qf krb5kdc[20632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577496, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:38:16.703518 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 20739
I20260928 06:38:16.703665 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:38:16.703944 5208 external_mini_cluster.cc:1428] Setting key d0936ba99c68658f680b9c780b075956
I20260928 06:38:16.707947 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:16.695797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:48683 (local address 127.5.22.62:43541)
0928 06:38:16.695954 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:16.695958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:16.696778 (+ 820us) server_negotiation.cc:407] Connection header received
0928 06:38:16.698118 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:16.698121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:16.698182 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:16.698310 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:16.699509 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:16.700056 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:16.701215 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:16.701458 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:16.701573 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:16.704094 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:16.704098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:16.704100 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:16.704123 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:16.705640 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:16.706254 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:16.706257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:16.706258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:16.706353 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:16.706746 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:16.706748 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:16.706749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:16.706918 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:16.707026 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:16.707595 (+ 569us) server_negotiation.cc:299] Negotiation successful
0928 06:38:16.707736 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 06:38:16.708660 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:16.696178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.1:48683)
0928 06:38:16.696629 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:16.696673 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:16.697904 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:16.698513 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:16.698521 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:16.698875 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:16.699345 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:16.699359 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:16.700187 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:16.700191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:16.701081 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:16.701091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:16.701618 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:16.701622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:16.701789 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:16.702364 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:16.702380 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:16.703921 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:16.705761 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:16.705770 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:16.705789 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:16.706150 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:16.706449 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:16.706455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:16.706460 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:16.706649 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:16.707031 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:16.707040 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:16.707349 (+ 309us) client_negotiation.cc:769] Sending connection context
0928 06:38:16.707511 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 06:38:16.707754 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":108,"threads_started":1}
I20260928 06:38:16.709995 20869 heartbeater.cc:347] Connected to a master server at 127.5.22.62:43541
I20260928 06:38:16.710263 20869 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:16.710914 20869 heartbeater.cc:511] Master 127.5.22.62:43541 requested a full tablet report, sending...
I20260928 06:38:16.712579 20677 ts_manager.cc:194] Registered new tserver with Master: 76f779950ca746809a2f124299de1c55 (127.5.22.1:34999)
I20260928 06:38:16.713663 20677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.1@KRBTEST.COM'} at 127.5.22.1:48683
WARNING: no policy specified for kudu/127.5.22.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
I20260928 06:38:16.734498 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:43541
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:38:16.843988 20874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:16.844273 20874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:16.844352 20874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:16.848287 20874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:16.848385 20874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:16.848505 20874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:38:16.853715 20874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:43541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:38:16.854864 20874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:16.855962 20874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:16.866096 20880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:16.866261 20874 server_base.cc:1034] running on GCE node
W20260928 06:38:16.866099 20879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:16.866434 20882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:16.866813 20874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:16.867456 20874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:16.868655 20874 hybrid_clock.cc:648] HybridClock initialized: now 1790577496868640 us; error 25 us; skew 500 ppm
Sep 28 06:38:16 dist-test-slave-j0qf krb5kdc[20632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577496, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:16.871609 20874 init.cc:377] Logged in from keytab as kudu/127.5.22.2@KRBTEST.COM (short username kudu)
I20260928 06:38:16.872511 20874 webserver.cc:460] Webserver started at http://127.5.22.2:33651/ using document root <none> and password file <none>
I20260928 06:38:16.873095 20874 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:16.873189 20874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:16.873433 20874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:38:16.875592 20874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "bd9107d3a54a4f998035e27180f4b8bc"
format_stamp: "Formatted at 2026-09-28 06:38:16 on dist-test-slave-j0qf"
server_key: "15044b77039649d331b6173c514aa563"
server_key_iv: "f3ef15d21f657fd8ba3d3f3b500291dc"
server_key_version: "encryptionkey@0"
I20260928 06:38:16.876178 20874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "bd9107d3a54a4f998035e27180f4b8bc"
format_stamp: "Formatted at 2026-09-28 06:38:16 on dist-test-slave-j0qf"
server_key: "15044b77039649d331b6173c514aa563"
server_key_iv: "f3ef15d21f657fd8ba3d3f3b500291dc"
server_key_version: "encryptionkey@0"
I20260928 06:38:16.879972 20874 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:38:16.882491 20889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:16.883802 20874 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:38:16.883939 20874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "bd9107d3a54a4f998035e27180f4b8bc"
format_stamp: "Formatted at 2026-09-28 06:38:16 on dist-test-slave-j0qf"
server_key: "15044b77039649d331b6173c514aa563"
server_key_iv: "f3ef15d21f657fd8ba3d3f3b500291dc"
server_key_version: "encryptionkey@0"
I20260928 06:38:16.884083 20874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:16.918370 20874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:16.921991 20874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:16.922211 20874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:16.922884 20874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:16.923766 20874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:38:16.923840 20874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:16.923913 20874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:38:16.923964 20874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:16.933550 20874 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:36871
I20260928 06:38:16.933629 21003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:36871 every 8 connection(s)
I20260928 06:38:16.934562 20874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:38:16.941160 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 20874
I20260928 06:38:16.941293 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:38:16.941632 5208 external_mini_cluster.cc:1428] Setting key 3f2e615d29bc63f91b9c3d167b608f49
Sep 28 06:38:16 dist-test-slave-j0qf krb5kdc[20632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577496, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:38:16.949438 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:16.936601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:36835 (local address 127.5.22.62:43541)
0928 06:38:16.936757 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:16.936762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:16.937926 (+ 1164us) server_negotiation.cc:407] Connection header received
0928 06:38:16.938846 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:16.938848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:16.938896 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:16.938971 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:16.940226 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:16.940815 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:16.942692 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:16.942930 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:16.943039 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:16.945880 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:16.945883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:16.945885 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:16.945914 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:16.947425 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:16.947944 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:16.947948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:16.947950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:16.948032 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:16.948388 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:16.948391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:16.948393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:16.948569 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:16.948679 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:16.949162 (+ 483us) server_negotiation.cc:299] Negotiation successful
0928 06:38:16.949277 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 06:38:16.950248 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:16.937012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.2:36835)
0928 06:38:16.937746 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:16.937815 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:16.938650 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:16.939158 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:16.939164 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:16.939548 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:16.940090 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:16.940104 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:16.940971 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:16.940977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:16.942572 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:16.942584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:16.943119 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:16.943124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:16.943331 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:16.944035 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:16.944054 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:16.945709 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:16.947550 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:16.947556 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:16.947570 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:16.947832 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:16.948135 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:16.948138 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:16.948140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:16.948285 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:16.948679 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:16.948687 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:16.948926 (+ 239us) client_negotiation.cc:769] Sending connection context
0928 06:38:16.949117 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 06:38:16.949444 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":104,"threads_started":1}
I20260928 06:38:16.951514 21004 heartbeater.cc:347] Connected to a master server at 127.5.22.62:43541
I20260928 06:38:16.951810 21004 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.22.3@KRBTEST.COM; defaulting to no policy
I20260928 06:38:16.952450 21004 heartbeater.cc:511] Master 127.5.22.62:43541 requested a full tablet report, sending...
I20260928 06:38:16.953591 20677 ts_manager.cc:194] Registered new tserver with Master: bd9107d3a54a4f998035e27180f4b8bc (127.5.22.2:36871)
I20260928 06:38:16.954152 20677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.2@KRBTEST.COM'} at 127.5.22.2:36835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
I20260928 06:38:16.973397 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:43541
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:38:17.087172 21009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:17.087484 21009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:17.087532 21009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:17.091861 21009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:17.091912 21009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:17.091997 21009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:38:17.097144 21009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:43541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:38:17.098168 21009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:17.099239 21009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:17.109259 21014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:17.109432 21009 server_base.cc:1034] running on GCE node
W20260928 06:38:17.109464 21017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:17.109320 21015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:17.109987 21009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:17.110723 21009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:17.111941 21009 hybrid_clock.cc:648] HybridClock initialized: now 1790577497111935 us; error 38 us; skew 500 ppm
Sep 28 06:38:17 dist-test-slave-j0qf krb5kdc[20632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577497, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:17.115046 21009 init.cc:377] Logged in from keytab as kudu/127.5.22.3@KRBTEST.COM (short username kudu)
I20260928 06:38:17.115913 21009 webserver.cc:460] Webserver started at http://127.5.22.3:38741/ using document root <none> and password file <none>
I20260928 06:38:17.116441 21009 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:17.116501 21009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:17.116729 21009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:38:17.118866 21009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "70347f0cca4f47d495db897103efcebb"
format_stamp: "Formatted at 2026-09-28 06:38:17 on dist-test-slave-j0qf"
server_key: "dd8417aa4784926e713612d360f64b4a"
server_key_iv: "6f23a2f84725c56e63a35f6189a99da3"
server_key_version: "encryptionkey@0"
I20260928 06:38:17.119450 21009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "70347f0cca4f47d495db897103efcebb"
format_stamp: "Formatted at 2026-09-28 06:38:17 on dist-test-slave-j0qf"
server_key: "dd8417aa4784926e713612d360f64b4a"
server_key_iv: "6f23a2f84725c56e63a35f6189a99da3"
server_key_version: "encryptionkey@0"
I20260928 06:38:17.123349 21009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 06:38:17.125861 21024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:17.127142 21009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 06:38:17.127269 21009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "70347f0cca4f47d495db897103efcebb"
format_stamp: "Formatted at 2026-09-28 06:38:17 on dist-test-slave-j0qf"
server_key: "dd8417aa4784926e713612d360f64b4a"
server_key_iv: "6f23a2f84725c56e63a35f6189a99da3"
server_key_version: "encryptionkey@0"
I20260928 06:38:17.127380 21009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:17.151490 21009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:17.154894 21009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:17.155117 21009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:17.155725 21009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:17.156555 21009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:38:17.156603 21009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:17.156672 21009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:38:17.156709 21009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:17.165843 21009 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:46097
I20260928 06:38:17.165884 21138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:46097 every 8 connection(s)
I20260928 06:38:17.166785 21009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:38:17.170680 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 21009
I20260928 06:38:17.170828 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
I20260928 06:38:17.171155 5208 external_mini_cluster.cc:1428] Setting key f7ae3d806daeb8445b1c38f94adc6160
Sep 28 06:38:17 dist-test-slave-j0qf krb5kdc[20632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577497, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:38:17.181212 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:17.168676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:39751 (local address 127.5.22.62:43541)
0928 06:38:17.168843 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:17.168848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:17.169685 (+ 837us) server_negotiation.cc:407] Connection header received
0928 06:38:17.171348 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:17.171352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:17.171423 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:17.171544 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:17.172834 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.173459 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.174792 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.175042 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.175164 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:17.177683 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:17.177686 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:17.177688 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:17.177710 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:17.179107 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:17.179691 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:17.179693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:17.179695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:17.179773 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:17.180224 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:17.180280 (+ 56us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:17.180282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:17.180454 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:17.180535 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:17.180985 (+ 450us) server_negotiation.cc:299] Negotiation successful
0928 06:38:17.181095 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 06:38:17.181996 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:17.169058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.3:39751)
0928 06:38:17.169534 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:17.169577 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:17.171076 (+ 1499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:17.171740 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:17.171747 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:17.172139 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:17.172675 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:17.172689 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.173608 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.173618 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:17.174658 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:17.174669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.175205 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.175209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:17.175402 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:17.176043 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:17.176060 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:17.177529 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:17.179215 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:17.179219 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:17.179230 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:17.179588 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:17.179879 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:17.179883 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:17.179885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:17.180132 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:17.180552 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:17.180557 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:17.180773 (+ 216us) client_negotiation.cc:769] Sending connection context
0928 06:38:17.180940 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 06:38:17.181178 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":101,"threads_started":1}
I20260928 06:38:17.183202 21139 heartbeater.cc:347] Connected to a master server at 127.5.22.62:43541
I20260928 06:38:17.183470 21139 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:17.184082 21139 heartbeater.cc:511] Master 127.5.22.62:43541 requested a full tablet report, sending...
I20260928 06:38:17.185134 20677 ts_manager.cc:194] Registered new tserver with Master: 70347f0cca4f47d495db897103efcebb (127.5.22.3:46097)
I20260928 06:38:17.185797 20677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.3@KRBTEST.COM'} at 127.5.22.3:39751
I20260928 06:38:17.196556 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:38:17.205426 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:17.198534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58824 (local address 127.5.22.62:43541)
0928 06:38:17.198677 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:17.198682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:17.198822 (+ 140us) server_negotiation.cc:407] Connection header received
0928 06:38:17.198934 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:17.198936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:17.198973 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:17.199033 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:17.199681 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.200184 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.201041 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.201248 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.201325 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:17.202733 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:17.202736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:17.202738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:17.202758 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:17.203988 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:17.204357 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:17.204359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:17.204361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:17.204467 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:17.204696 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:17.204698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:17.204699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:17.204901 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:17.205011 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:17.205189 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 06:38:17.205277 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:38:17.208168 20677 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:38:17.212002 20677 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:38:17.226169 21151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:17.220236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60872 (local address 127.5.22.1:34999)
0928 06:38:17.220946 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:17.220956 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:17.220984 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:38:17.221066 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:17.221073 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:17.221280 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:17.221511 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:17.221830 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.223059 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.224765 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.225430 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.225535 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:17.225680 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:17.225900 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 06:38:17.226018 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":124,"threads_started":1}
I20260928 06:38:17.226150 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:17.220297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:36871 (local address 127.0.0.1:43382)
0928 06:38:17.221023 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:17.221037 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:17.221133 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:17.222029 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:17.222034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:17.222050 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:17.222167 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:17.222175 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.223498 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.223501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:17.224663 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:17.224674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.225834 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.225837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:17.225917 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:17.225937 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 06:38:17.226002 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:38:17.226055 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"mutex_wait_us":17,"thread_start_us":135,"threads_started":1}
I20260928 06:38:17.226147 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:17.220090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:34999 (local address 127.0.0.1:60872)
0928 06:38:17.220460 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:17.220507 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:17.220631 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:17.221503 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:17.221509 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:17.221535 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:17.221684 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:17.221690 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.223183 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.223186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:17.224626 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:17.224636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.225560 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.225567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:17.225685 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:17.225725 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 06:38:17.225840 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 06:38:17.225980 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":56,"thread_start_us":107,"threads_started":1}
I20260928 06:38:17.226673 21154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:17.220969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.5.22.2:36871)
0928 06:38:17.221478 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:17.221489 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:17.221510 (+ 21us) server_negotiation.cc:407] Connection header received
0928 06:38:17.221578 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:17.221582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:17.221764 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:17.221926 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:17.222284 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.223362 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.224811 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.225528 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.225630 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:17.225768 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:17.226410 (+ 642us) server_negotiation.cc:299] Negotiation successful
0928 06:38:17.226518 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":113,"threads_started":1}
I20260928 06:38:17.226686 21153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:17.220854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:46097 (local address 127.0.0.1:58796)
0928 06:38:17.221968 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:17.221983 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:17.222061 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:17.222589 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:17.222592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:17.222609 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:17.222696 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:17.222701 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.223928 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.223931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:17.225055 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:17.225065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.226225 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.226231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:17.226389 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:17.226422 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 06:38:17.226496 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 06:38:17.226560 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1053,"mutex_wait_us":21,"thread_start_us":95,"threads_started":1}
I20260928 06:38:17.227073 21155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:17.221269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58796 (local address 127.5.22.3:46097)
0928 06:38:17.221828 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:17.221840 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:17.222154 (+ 314us) server_negotiation.cc:407] Connection header received
0928 06:38:17.222235 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:17.222242 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:17.222462 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:17.222608 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:17.222790 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.223783 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.225181 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.225944 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.226041 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:17.226156 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:17.226842 (+ 686us) server_negotiation.cc:299] Negotiation successful
0928 06:38:17.226935 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":116,"threads_started":1}
I20260928 06:38:17.229332 20803 tablet_service.cc:1467] Processing CreateTablet for tablet b54a6f3886b2419eb9ca339d094eec20 (DEFAULT_TABLE table=test-table [id=92e4bee17b954ca58c68d7ec0b7a83ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:38:17.230418 20803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54a6f3886b2419eb9ca339d094eec20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:17.231976 21073 tablet_service.cc:1467] Processing CreateTablet for tablet b54a6f3886b2419eb9ca339d094eec20 (DEFAULT_TABLE table=test-table [id=92e4bee17b954ca58c68d7ec0b7a83ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:38:17.232334 20938 tablet_service.cc:1467] Processing CreateTablet for tablet b54a6f3886b2419eb9ca339d094eec20 (DEFAULT_TABLE table=test-table [id=92e4bee17b954ca58c68d7ec0b7a83ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:38:17.233011 21073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54a6f3886b2419eb9ca339d094eec20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:17.233273 20938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54a6f3886b2419eb9ca339d094eec20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:17.237924 21156 tablet_bootstrap.cc:492] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55: Bootstrap starting.
I20260928 06:38:17.239383 21157 tablet_bootstrap.cc:492] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc: Bootstrap starting.
I20260928 06:38:17.240156 21158 tablet_bootstrap.cc:492] T b54a6f3886b2419eb9ca339d094eec20 P 70347f0cca4f47d495db897103efcebb: Bootstrap starting.
I20260928 06:38:17.240408 21156 tablet_bootstrap.cc:654] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:17.241333 21156 log.cc:824] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:17.242273 21157 tablet_bootstrap.cc:654] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:17.242276 21158 tablet_bootstrap.cc:654] T b54a6f3886b2419eb9ca339d094eec20 P 70347f0cca4f47d495db897103efcebb: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:17.243253 21157 log.cc:824] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:17.243321 21158 log.cc:824] T b54a6f3886b2419eb9ca339d094eec20 P 70347f0cca4f47d495db897103efcebb: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:17.244076 21156 tablet_bootstrap.cc:492] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55: No bootstrap required, opened a new log
I20260928 06:38:17.244268 21156 ts_tablet_manager.cc:1397] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 06:38:17.245205 21157 tablet_bootstrap.cc:492] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc: No bootstrap required, opened a new log
I20260928 06:38:17.245204 21158 tablet_bootstrap.cc:492] T b54a6f3886b2419eb9ca339d094eec20 P 70347f0cca4f47d495db897103efcebb: No bootstrap required, opened a new log
I20260928 06:38:17.245373 21157 ts_tablet_manager.cc:1397] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 06:38:17.245373 21158 ts_tablet_manager.cc:1397] T b54a6f3886b2419eb9ca339d094eec20 P 70347f0cca4f47d495db897103efcebb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 06:38:17.247656 21156 raft_consensus.cc:348] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } } peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } }
I20260928 06:38:17.247896 21156 raft_consensus.cc:374] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:17.247969 21156 raft_consensus.cc:729] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f779950ca746809a2f124299de1c55, State: Initialized, Role: FOLLOWER
I20260928 06:38:17.248426 21156 consensus_queue.cc:260] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } } peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } }
I20260928 06:38:17.249233 21158 raft_consensus.cc:348] T b54a6f3886b2419eb9ca339d094eec20 P 70347f0cca4f47d495db897103efcebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } } peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } }
I20260928 06:38:17.249233 21157 raft_consensus.cc:348] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } } peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } }
I20260928 06:38:17.249500 21156 ts_tablet_manager.cc:1428] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 06:38:17.249504 21157 raft_consensus.cc:374] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:17.249547 21158 raft_consensus.cc:374] T b54a6f3886b2419eb9ca339d094eec20 P 70347f0cca4f47d495db897103efcebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:17.249562 21157 raft_consensus.cc:729] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd9107d3a54a4f998035e27180f4b8bc, State: Initialized, Role: FOLLOWER
I20260928 06:38:17.249648 21158 raft_consensus.cc:729] T b54a6f3886b2419eb9ca339d094eec20 P 70347f0cca4f47d495db897103efcebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70347f0cca4f47d495db897103efcebb, State: Initialized, Role: FOLLOWER
I20260928 06:38:17.249917 20869 heartbeater.cc:503] Master 127.5.22.62:43541 was elected leader, sending a full tablet report...
I20260928 06:38:17.250048 21157 consensus_queue.cc:260] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } } peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } }
I20260928 06:38:17.250048 21158 consensus_queue.cc:260] T b54a6f3886b2419eb9ca339d094eec20 P 70347f0cca4f47d495db897103efcebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } } peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } }
I20260928 06:38:17.251026 21157 ts_tablet_manager.cc:1428] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260928 06:38:17.251026 21158 ts_tablet_manager.cc:1428] T b54a6f3886b2419eb9ca339d094eec20 P 70347f0cca4f47d495db897103efcebb: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260928 06:38:17.251433 21139 heartbeater.cc:503] Master 127.5.22.62:43541 was elected leader, sending a full tablet report...
I20260928 06:38:17.251587 21004 heartbeater.cc:503] Master 127.5.22.62:43541 was elected leader, sending a full tablet report...
I20260928 06:38:17.338620 21163 raft_consensus.cc:482] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:38:17.338876 21163 raft_consensus.cc:504] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } } peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } }
I20260928 06:38:17.340145 21163 leader_election.cc:288] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76f779950ca746809a2f124299de1c55 (127.5.22.1:34999), 70347f0cca4f47d495db897103efcebb (127.5.22.3:46097)
I20260928 06:38:17.343990 21151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:17.340564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:33101 (local address 127.5.22.1:34999)
0928 06:38:17.340723 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:17.340729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:17.340744 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:38:17.340803 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:17.340826 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:17.340908 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:17.340987 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:17.341308 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.341833 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.343028 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.343436 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.343559 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:17.343626 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:17.343803 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 06:38:17.343886 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 06:38:17.343989 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:17.340389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:34999 (local address 127.5.22.2:33101)
0928 06:38:17.340548 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:17.340564 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:17.340670 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:17.341040 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:17.341044 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:17.341061 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:17.341176 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:17.341181 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.341950 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.341957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:17.342903 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:17.342913 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.343557 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.343560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:17.343669 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:17.343696 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 06:38:17.343785 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:38:17.343857 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 06:38:17.344323 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:17.340674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:46097 (local address 127.5.22.2:55063)
0928 06:38:17.341000 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:17.341015 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:17.341137 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:17.341406 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:17.341409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:17.341425 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:17.341549 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:17.341555 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.342258 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.342261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:17.343206 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:17.343216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.343876 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.343879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:17.343988 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:17.344012 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 06:38:17.344113 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 06:38:17.344197 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":171,"threads_started":1}
I20260928 06:38:17.344408 21155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:17.340797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:55063 (local address 127.5.22.3:46097)
0928 06:38:17.340970 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:17.340976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:17.341215 (+ 239us) server_negotiation.cc:407] Connection header received
0928 06:38:17.341264 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:17.341266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:17.341306 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:17.341392 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:17.341658 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.342141 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.343332 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.343751 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.343872 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:17.343929 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:17.344214 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 06:38:17.344287 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 06:38:17.344718 20823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54a6f3886b2419eb9ca339d094eec20" candidate_uuid: "bd9107d3a54a4f998035e27180f4b8bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76f779950ca746809a2f124299de1c55" is_pre_election: true
I20260928 06:38:17.345054 20823 raft_consensus.cc:2454] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd9107d3a54a4f998035e27180f4b8bc in term 0.
I20260928 06:38:17.345060 21093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54a6f3886b2419eb9ca339d094eec20" candidate_uuid: "bd9107d3a54a4f998035e27180f4b8bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70347f0cca4f47d495db897103efcebb" is_pre_election: true
I20260928 06:38:17.345340 21093 raft_consensus.cc:2454] T b54a6f3886b2419eb9ca339d094eec20 P 70347f0cca4f47d495db897103efcebb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd9107d3a54a4f998035e27180f4b8bc in term 0.
I20260928 06:38:17.345623 20893 leader_election.cc:302] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76f779950ca746809a2f124299de1c55, bd9107d3a54a4f998035e27180f4b8bc; no voters:
I20260928 06:38:17.345935 21163 raft_consensus.cc:2781] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:38:17.346009 21163 raft_consensus.cc:482] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:38:17.346038 21163 raft_consensus.cc:3037] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:17.347100 21163 raft_consensus.cc:504] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } } peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } }
I20260928 06:38:17.347438 21163 leader_election.cc:288] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc [CANDIDATE]: Term 1 election: Requested vote from peers 76f779950ca746809a2f124299de1c55 (127.5.22.1:34999), 70347f0cca4f47d495db897103efcebb (127.5.22.3:46097)
I20260928 06:38:17.347839 20823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54a6f3886b2419eb9ca339d094eec20" candidate_uuid: "bd9107d3a54a4f998035e27180f4b8bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76f779950ca746809a2f124299de1c55"
I20260928 06:38:17.347889 21093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54a6f3886b2419eb9ca339d094eec20" candidate_uuid: "bd9107d3a54a4f998035e27180f4b8bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70347f0cca4f47d495db897103efcebb"
I20260928 06:38:17.347976 20823 raft_consensus.cc:3037] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:17.348002 21093 raft_consensus.cc:3037] T b54a6f3886b2419eb9ca339d094eec20 P 70347f0cca4f47d495db897103efcebb [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:17.349018 21093 raft_consensus.cc:2454] T b54a6f3886b2419eb9ca339d094eec20 P 70347f0cca4f47d495db897103efcebb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd9107d3a54a4f998035e27180f4b8bc in term 1.
I20260928 06:38:17.349049 20823 raft_consensus.cc:2454] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd9107d3a54a4f998035e27180f4b8bc in term 1.
I20260928 06:38:17.349421 20893 leader_election.cc:302] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76f779950ca746809a2f124299de1c55, bd9107d3a54a4f998035e27180f4b8bc; no voters:
I20260928 06:38:17.349596 21163 raft_consensus.cc:2781] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:38:17.349869 21163 raft_consensus.cc:686] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc [term 1 LEADER]: Becoming Leader. State: Replica: bd9107d3a54a4f998035e27180f4b8bc, State: Running, Role: LEADER
I20260928 06:38:17.350220 21163 consensus_queue.cc:237] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } } peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } }
I20260928 06:38:17.353446 20676 catalog_manager.cc:5495] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc reported cstate change: term changed from 0 to 1, leader changed from <none> to bd9107d3a54a4f998035e27180f4b8bc (127.5.22.2). New cstate: current_term: 1 leader_uuid: "bd9107d3a54a4f998035e27180f4b8bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } health_report { overall_health: HEALTHY } } }
I20260928 06:38:17.379118 21154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:17.374626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43384 (local address 127.5.22.2:36871)
0928 06:38:17.374741 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:17.374745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:17.374823 (+ 78us) server_negotiation.cc:407] Connection header received
0928 06:38:17.374987 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:17.374990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:17.375048 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:17.375134 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:38:17.375397 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.375885 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.377145 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:17.377355 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:17.377432 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:17.377814 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:38:17.378435 (+ 621us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:38:17.378563 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:17.378904 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 06:38:17.378977 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:38:17.385056 21093 raft_consensus.cc:1261] T b54a6f3886b2419eb9ca339d094eec20 P 70347f0cca4f47d495db897103efcebb [term 1 FOLLOWER]: Refusing update from remote peer bd9107d3a54a4f998035e27180f4b8bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:38:17.385056 20823 raft_consensus.cc:1261] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55 [term 1 FOLLOWER]: Refusing update from remote peer bd9107d3a54a4f998035e27180f4b8bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:38:17.385753 21163 consensus_queue.cc:1035] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:38:17.386015 21166 consensus_queue.cc:1035] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:38:17.394346 21169 mvcc.cc:205] Tried to move back new op lower bound from 7334205429282000896 to 7334205429149450240. Current Snapshot: MvccSnapshot[applied={T|T < 7334205429282000896}]
I20260928 06:38:17.394737 21170 mvcc.cc:205] Tried to move back new op lower bound from 7334205429282000896 to 7334205429149450240. Current Snapshot: MvccSnapshot[applied={T|T < 7334205429282000896}]
I20260928 06:38:17.402666 20673 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:38:17.402849 20673 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:38:17.405267 20673 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194: Sending DeleteTablet for 3 replicas of tablet b54a6f3886b2419eb9ca339d094eec20
I20260928 06:38:17.406129 20938 tablet_service.cc:1514] Processing DeleteTablet for tablet b54a6f3886b2419eb9ca339d094eec20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:38:17 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:43382
I20260928 06:38:17.406138 20803 tablet_service.cc:1514] Processing DeleteTablet for tablet b54a6f3886b2419eb9ca339d094eec20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:38:17 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:60872
I20260928 06:38:17.406299 21073 tablet_service.cc:1514] Processing DeleteTablet for tablet b54a6f3886b2419eb9ca339d094eec20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:38:17 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:58796
I20260928 06:38:17.406766 21179 tablet_replica.cc:330] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc: stopping tablet replica
I20260928 06:38:17.406911 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 20739
I20260928 06:38:17.407054 21178 tablet_replica.cc:330] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55: stopping tablet replica
I20260928 06:38:17.407133 21179 raft_consensus.cc:2229] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:38:17.407307 21178 raft_consensus.cc:2229] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:38:17.407620 21180 tablet_replica.cc:330] T b54a6f3886b2419eb9ca339d094eec20 P 70347f0cca4f47d495db897103efcebb: stopping tablet replica
I20260928 06:38:17.407641 21179 raft_consensus.cc:2258] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:38:17.407924 21180 raft_consensus.cc:2229] T b54a6f3886b2419eb9ca339d094eec20 P 70347f0cca4f47d495db897103efcebb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:38:17.408242 21180 raft_consensus.cc:2258] T b54a6f3886b2419eb9ca339d094eec20 P 70347f0cca4f47d495db897103efcebb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:38:17.409288 21179 ts_tablet_manager.cc:1905] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:38:17.409709 21180 ts_tablet_manager.cc:1905] T b54a6f3886b2419eb9ca339d094eec20 P 70347f0cca4f47d495db897103efcebb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:38:17.412202 21179 ts_tablet_manager.cc:1918] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:38:17.412340 21180 ts_tablet_manager.cc:1918] T b54a6f3886b2419eb9ca339d094eec20 P 70347f0cca4f47d495db897103efcebb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:38:17.412374 21179 log.cc:1196] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/wal/wals/b54a6f3886b2419eb9ca339d094eec20
I20260928 06:38:17.412541 21180 log.cc:1196] T b54a6f3886b2419eb9ca339d094eec20 P 70347f0cca4f47d495db897103efcebb: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/b54a6f3886b2419eb9ca339d094eec20
I20260928 06:38:17.412734 21179 ts_tablet_manager.cc:1939] T b54a6f3886b2419eb9ca339d094eec20 P bd9107d3a54a4f998035e27180f4b8bc: Deleting consensus metadata
I20260928 06:38:17.412946 21180 ts_tablet_manager.cc:1939] T b54a6f3886b2419eb9ca339d094eec20 P 70347f0cca4f47d495db897103efcebb: Deleting consensus metadata
I20260928 06:38:17.414029 20664 catalog_manager.cc:4841] TS 70347f0cca4f47d495db897103efcebb (127.5.22.3:46097): tablet b54a6f3886b2419eb9ca339d094eec20 (table test-table [id=92e4bee17b954ca58c68d7ec0b7a83ca]) successfully deleted
I20260928 06:38:17.414110 20661 catalog_manager.cc:4841] TS bd9107d3a54a4f998035e27180f4b8bc (127.5.22.2:36871): tablet b54a6f3886b2419eb9ca339d094eec20 (table test-table [id=92e4bee17b954ca58c68d7ec0b7a83ca]) successfully deleted
W20260928 06:38:17.416704 20664 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.22.1:34999 (error 108)
I20260928 06:38:17.417582 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 20874
I20260928 06:38:17.417816 21153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:17.417507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:34999 (local address 127.0.0.1:60878)
0928 06:38:17.417669 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:17.417735 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.1:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260928 06:38:17.422474 20664 catalog_manager.cc:4584] TS 76f779950ca746809a2f124299de1c55 (127.5.22.1:34999): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b54a6f3886b2419eb9ca339d094eec20: Network error: Client connection negotiation failed: client connection to 127.5.22.1:34999: connect: Connection refused (error 111)
I20260928 06:38:17.423265 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 21009
I20260928 06:38:17.429141 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 20645
I20260928 06:38:37.435853 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:43541
--webserver_interface=127.5.22.62
--webserver_port=41199
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:43541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:38:37.547674 21182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:37.548053 21182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:37.548099 21182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:37.552286 21182 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 06:38:37.552347 21182 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 06:38:37.552364 21182 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 06:38:37.552378 21182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:38:37.552395 21182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:37.552413 21182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:37.552429 21182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:38:37.552444 21182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:38:37.557658 21182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:43541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:43541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=41199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:38:37.558699 21182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:37.559819 21182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:37.574429 21188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:37.574460 21189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:37.574458 21182 server_base.cc:1034] running on GCE node
W20260928 06:38:37.574565 21191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:37.575292 21182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:37.576511 21182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:37.577687 21182 hybrid_clock.cc:648] HybridClock initialized: now 1790577517577671 us; error 33 us; skew 500 ppm
Sep 28 06:38:37 dist-test-slave-j0qf krb5kdc[20632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577517, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:37.580519 21182 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:38:37.581452 21182 webserver.cc:460] Webserver started at http://127.5.22.62:41199/ using document root <none> and password file <none>
I20260928 06:38:37.582067 21182 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:37.582141 21182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:37.586074 21182 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:38:37.588356 21198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:37.589733 21182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 06:38:37.589872 21182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "a58cb30fef744b048351d0c17588c194"
format_stamp: "Formatted at 2026-09-28 06:38:16 on dist-test-slave-j0qf"
server_key: "921bad6934aa05c44df053c02f840b14"
server_key_iv: "afb617f63704388a57903759b7fea2ea"
server_key_version: "encryptionkey@0"
I20260928 06:38:37.590598 21182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:37.617924 21182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:37.621212 21182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:37.621416 21182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:37.628860 21182 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:43541
I20260928 06:38:37.629153 21252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:43541 every 8 connection(s)
I20260928 06:38:37.629992 21182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:38:37.632823 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 21182
I20260928 06:38:37.633289 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:34999
--local_ip_for_outbound_sockets=127.5.22.1
--tserver_master_addrs=127.5.22.62:43541
--webserver_port=40357
--webserver_interface=127.5.22.1
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:38:37.634690 21253 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:38:37.636375 21253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194: Bootstrap starting.
I20260928 06:38:37.654601 21253 log.cc:824] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:37.663345 21253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:38:37.663744 21253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194: Bootstrap complete.
I20260928 06:38:37.667881 21253 raft_consensus.cc:348] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58cb30fef744b048351d0c17588c194" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43541 } }
I20260928 06:38:37.668285 21253 raft_consensus.cc:729] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a58cb30fef744b048351d0c17588c194, State: Initialized, Role: FOLLOWER
I20260928 06:38:37.668752 21253 consensus_queue.cc:260] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58cb30fef744b048351d0c17588c194" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43541 } }
I20260928 06:38:37.668887 21253 raft_consensus.cc:388] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:38:37.668972 21253 raft_consensus.cc:482] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:38:37.669090 21253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:38:37.670997 21253 raft_consensus.cc:504] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58cb30fef744b048351d0c17588c194" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43541 } }
I20260928 06:38:37.671319 21253 leader_election.cc:302] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a58cb30fef744b048351d0c17588c194; no voters:
I20260928 06:38:37.671615 21253 leader_election.cc:288] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:38:37.671761 21257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:38:37.672005 21257 raft_consensus.cc:686] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [term 2 LEADER]: Becoming Leader. State: Replica: a58cb30fef744b048351d0c17588c194, State: Running, Role: LEADER
I20260928 06:38:37.672389 21257 consensus_queue.cc:237] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58cb30fef744b048351d0c17588c194" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43541 } }
I20260928 06:38:37.672844 21253 sys_catalog.cc:564] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:38:37.673892 21259 sys_catalog.cc:455] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a58cb30fef744b048351d0c17588c194. Latest consensus state: current_term: 2 leader_uuid: "a58cb30fef744b048351d0c17588c194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58cb30fef744b048351d0c17588c194" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43541 } } }
I20260928 06:38:37.674015 21259 sys_catalog.cc:458] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [sys.catalog]: This master's current role is: LEADER
I20260928 06:38:37.674403 21258 sys_catalog.cc:455] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a58cb30fef744b048351d0c17588c194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58cb30fef744b048351d0c17588c194" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43541 } } }
I20260928 06:38:37.674515 21258 sys_catalog.cc:458] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [sys.catalog]: This master's current role is: LEADER
I20260928 06:38:37.675109 21266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:38:37.679714 21266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:38:37.680348 21266 catalog_manager.cc:1259] Loaded cluster ID: a93ffe9a51214f6b97f899075cd011ea
I20260928 06:38:37.680409 21266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:38:37.682902 21266 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:38:37.683629 21266 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194: Loaded TSK: 0
I20260928 06:38:37.708047 21266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194: Generated new TSK 1
I20260928 06:38:37.708382 21266 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:38:37.753574 21255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:37.753885 21255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:37.753963 21255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:37.758065 21255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:37.758159 21255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:37.758253 21255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:38:37.763733 21255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:34999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=40357
--tserver_master_addrs=127.5.22.62:43541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:38:37.764982 21255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:37.766077 21255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:37.776405 21280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:37.776482 21281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:37.776646 21255 server_base.cc:1034] running on GCE node
W20260928 06:38:37.776917 21283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:37.777357 21255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:37.777990 21255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:37.779207 21255 hybrid_clock.cc:648] HybridClock initialized: now 1790577517779186 us; error 39 us; skew 500 ppm
Sep 28 06:38:37 dist-test-slave-j0qf krb5kdc[20632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577517, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:37.782054 21255 init.cc:377] Logged in from keytab as kudu/127.5.22.1@KRBTEST.COM (short username kudu)
I20260928 06:38:37.782974 21255 webserver.cc:460] Webserver started at http://127.5.22.1:40357/ using document root <none> and password file <none>
I20260928 06:38:37.783577 21255 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:37.783689 21255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:37.787693 21255 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:38:37.790135 21290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:37.791493 21255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:38:37.791631 21255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "76f779950ca746809a2f124299de1c55"
format_stamp: "Formatted at 2026-09-28 06:38:16 on dist-test-slave-j0qf"
server_key: "fab94183b6424fa54221b652212d737c"
server_key_iv: "1287890ecc8c241a725790426808791a"
server_key_version: "encryptionkey@0"
I20260928 06:38:37.792200 21255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:37.806970 21255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:37.810297 21255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:37.810564 21255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:37.811224 21255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:37.812340 21297 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:38:37.817797 21255 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:38:37.817891 21255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260928 06:38:37.817972 21255 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:38:37.819716 21255 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:38:37.819787 21255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260928 06:38:37.820115 21297 tablet_bootstrap.cc:492] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55: Bootstrap starting.
I20260928 06:38:37.830745 21255 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:34999
I20260928 06:38:37.830921 21406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:34999 every 8 connection(s)
I20260928 06:38:37.831916 21255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:38:37.840490 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 21255
I20260928 06:38:37.841032 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:36871
--local_ip_for_outbound_sockets=127.5.22.2
--tserver_master_addrs=127.5.22.62:43541
--webserver_port=33651
--webserver_interface=127.5.22.2
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:38:37.842037 21297 log.cc:824] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55: Log is configured to *not* fsync() on all Append() calls
Sep 28 06:38:37 dist-test-slave-j0qf krb5kdc[20632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577517, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:38:37.849455 21410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:37.834038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.1:45471)
0928 06:38:37.834593 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:37.834647 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:37.835563 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:37.836677 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:37.836685 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:37.837110 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:37.837574 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:37.837588 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:37.838950 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:37.838953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:37.839840 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:37.839851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:37.841043 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:37.841050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:37.841302 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:37.842078 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:37.842106 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:37.843874 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:37.846260 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:37.846266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:37.846279 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:37.846625 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:37.846982 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:37.846985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:37.846987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:37.847121 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:37.847710 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:37.847716 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:37.847982 (+ 266us) client_negotiation.cc:769] Sending connection context
0928 06:38:37.848161 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 06:38:37.848514 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":131,"threads_started":1}
I20260928 06:38:37.849442 21409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:37.834004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:45471 (local address 127.5.22.62:43541)
0928 06:38:37.835160 (+ 1156us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:37.835178 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:37.835223 (+ 45us) server_negotiation.cc:407] Connection header received
0928 06:38:37.835987 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:37.835997 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:37.836288 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:37.836620 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:37.837749 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:37.838789 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:37.839970 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:37.840636 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:37.840807 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:37.844135 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:37.844142 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:37.844160 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:37.844194 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:37.846139 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:37.846752 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:37.846758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:37.846761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:37.846878 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:37.847229 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:37.847233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:37.847234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:37.847594 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:37.847738 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:37.848294 (+ 556us) server_negotiation.cc:299] Negotiation successful
0928 06:38:37.848567 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1021,"thread_start_us":114,"threads_started":1}
I20260928 06:38:37.852240 21407 heartbeater.cc:347] Connected to a master server at 127.5.22.62:43541
I20260928 06:38:37.852492 21407 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:37.852977 21407 heartbeater.cc:511] Master 127.5.22.62:43541 requested a full tablet report, sending...
I20260928 06:38:37.853199 21297 tablet_bootstrap.cc:492] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:38:37.853688 21297 tablet_bootstrap.cc:492] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55: Bootstrap complete.
I20260928 06:38:37.855266 21297 ts_tablet_manager.cc:1397] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55: Time spent bootstrapping tablet: real 0.035s user 0.021s sys 0.012s
I20260928 06:38:37.855381 21215 ts_manager.cc:194] Registered new tserver with Master: 76f779950ca746809a2f124299de1c55 (127.5.22.1:34999)
I20260928 06:38:37.858246 21297 raft_consensus.cc:348] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } } peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } }
I20260928 06:38:37.858909 21297 raft_consensus.cc:729] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f779950ca746809a2f124299de1c55, State: Initialized, Role: FOLLOWER
I20260928 06:38:37.858955 21215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.1@KRBTEST.COM'} at 127.5.22.1:45471
I20260928 06:38:37.859457 21297 consensus_queue.cc:260] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } } peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } }
I20260928 06:38:37.860528 21297 ts_tablet_manager.cc:1428] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 06:38:37.861348 21407 heartbeater.cc:503] Master 127.5.22.62:43541 was elected leader, sending a full tablet report...
I20260928 06:38:37.865984 21413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:37.858228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:34999 (local address 127.0.0.1:52992)
0928 06:38:37.859510 (+ 1282us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:37.859554 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:37.859678 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:37.861859 (+ 2181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:37.861866 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:37.861894 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:37.862027 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:37.862033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:37.863291 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:37.863295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:37.864607 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:37.864617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:37.865441 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:37.865445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:37.865554 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:37.865590 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 06:38:37.865704 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 06:38:37.865836 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1117,"thread_start_us":152,"threads_started":1}
I20260928 06:38:37.865984 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:37.859020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52992 (local address 127.5.22.1:34999)
0928 06:38:37.861009 (+ 1989us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:37.861020 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:37.861043 (+ 23us) server_negotiation.cc:407] Connection header received
0928 06:38:37.861110 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:37.861117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:37.861329 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:37.861526 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:37.862155 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:37.863180 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:37.864745 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:37.865311 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:37.865438 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:37.865587 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:37.865739 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 06:38:37.865832 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1900,"thread_start_us":112,"threads_started":1}
I20260928 06:38:37.867094 21340 tablet_service.cc:1514] Processing DeleteTablet for tablet b54a6f3886b2419eb9ca339d094eec20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:38:17 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:52992
I20260928 06:38:37.867940 21416 tablet_replica.cc:330] stopping tablet replica
I20260928 06:38:37.868266 21416 raft_consensus.cc:2229] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:38:37.868536 21416 raft_consensus.cc:2258] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:38:37.869655 21416 ts_tablet_manager.cc:1905] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:38:37.874473 21416 ts_tablet_manager.cc:1918] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:38:37.874589 21416 log.cc:1196] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/wal/wals/b54a6f3886b2419eb9ca339d094eec20
I20260928 06:38:37.874877 21416 ts_tablet_manager.cc:1939] T b54a6f3886b2419eb9ca339d094eec20 P 76f779950ca746809a2f124299de1c55: Deleting consensus metadata
I20260928 06:38:37.876941 21202 catalog_manager.cc:4841] TS 76f779950ca746809a2f124299de1c55 (127.5.22.1:34999): tablet b54a6f3886b2419eb9ca339d094eec20 (table test-table [id=92e4bee17b954ca58c68d7ec0b7a83ca]) successfully deleted
W20260928 06:38:37.964450 21411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:37.964740 21411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:37.964788 21411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:37.969077 21411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:37.969144 21411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:37.969242 21411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:38:37.975088 21411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:36871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=33651
--tserver_master_addrs=127.5.22.62:43541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:38:37.976171 21411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:37.977195 21411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:37.986806 21421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:37.986824 21424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:37.986824 21422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:37.986929 21411 server_base.cc:1034] running on GCE node
I20260928 06:38:37.987522 21411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:37.988217 21411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:37.989410 21411 hybrid_clock.cc:648] HybridClock initialized: now 1790577517989395 us; error 28 us; skew 500 ppm
Sep 28 06:38:37 dist-test-slave-j0qf krb5kdc[20632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577517, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:37.992184 21411 init.cc:377] Logged in from keytab as kudu/127.5.22.2@KRBTEST.COM (short username kudu)
I20260928 06:38:37.993064 21411 webserver.cc:460] Webserver started at http://127.5.22.2:33651/ using document root <none> and password file <none>
I20260928 06:38:37.993661 21411 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:37.993760 21411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:37.997911 21411 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:38:38.000125 21431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:38.001417 21411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:38:38.001567 21411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "bd9107d3a54a4f998035e27180f4b8bc"
format_stamp: "Formatted at 2026-09-28 06:38:16 on dist-test-slave-j0qf"
server_key: "15044b77039649d331b6173c514aa563"
server_key_iv: "f3ef15d21f657fd8ba3d3f3b500291dc"
server_key_version: "encryptionkey@0"
I20260928 06:38:38.002151 21411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:38.022703 21411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:38.026129 21411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:38.026389 21411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:38.027180 21411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:38.028055 21411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:38:38.028126 21411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:38.028198 21411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:38:38.028249 21411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:38.037277 21411 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:36871
I20260928 06:38:38.037321 21545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:36871 every 8 connection(s)
I20260928 06:38:38.038250 21411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:38:38.039089 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 21411
I20260928 06:38:38.039502 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:46097
--local_ip_for_outbound_sockets=127.5.22.3
--tserver_master_addrs=127.5.22.62:43541
--webserver_port=38741
--webserver_interface=127.5.22.3
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:38:38 dist-test-slave-j0qf krb5kdc[20632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577517, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:38:38.054862 21409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:38.040129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:35335 (local address 127.5.22.62:43541)
0928 06:38:38.040317 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:38.040321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:38.041972 (+ 1651us) server_negotiation.cc:407] Connection header received
0928 06:38:38.042924 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:38.042927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:38.042981 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:38.043092 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:38.044800 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.045462 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.047424 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.047667 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.047762 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:38.051003 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:38.051006 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:38.051008 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:38.051036 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:38.052632 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:38.053207 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:38.053209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:38.053211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:38.053281 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:38.053636 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:38.053638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:38.053639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:38.053833 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:38.053909 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:38.054513 (+ 604us) server_negotiation.cc:299] Negotiation successful
0928 06:38:38.054623 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:38:38.055853 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:38.040646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.2:35335)
0928 06:38:38.041766 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:38.041819 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:38.042707 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:38.043305 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:38.043317 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:38.043855 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:38.044579 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:38.044603 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.045613 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.045618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:38.047254 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:38.047267 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.047836 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.047844 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:38.048093 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:38.048772 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:38.048801 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:38.050782 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:38.052747 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:38.052753 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:38.052769 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:38.053104 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:38.053371 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:38.053374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:38.053376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:38.053545 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:38.053928 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:38.053934 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:38.054231 (+ 297us) client_negotiation.cc:769] Sending connection context
0928 06:38:38.054454 (+ 223us) client_negotiation.cc:240] Negotiation successful
0928 06:38:38.054786 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904,"thread_start_us":706,"threads_started":1}
I20260928 06:38:38.057073 21546 heartbeater.cc:347] Connected to a master server at 127.5.22.62:43541
I20260928 06:38:38.057403 21546 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:38.057899 21546 heartbeater.cc:511] Master 127.5.22.62:43541 requested a full tablet report, sending...
I20260928 06:38:38.058982 21215 ts_manager.cc:194] Registered new tserver with Master: bd9107d3a54a4f998035e27180f4b8bc (127.5.22.2:36871)
I20260928 06:38:38.059638 21215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.2@KRBTEST.COM'} at 127.5.22.2:35335
W20260928 06:38:38.156364 21548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:38.156612 21548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:38.156654 21548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:38.160728 21548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:38.160789 21548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:38.160885 21548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:38:38.165987 21548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:46097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=38741
--tserver_master_addrs=127.5.22.62:43541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:38:38.167034 21548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:38.168020 21548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:38.177759 21555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:38.177874 21554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:38.177759 21557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:38.178030 21548 server_base.cc:1034] running on GCE node
I20260928 06:38:38.178627 21548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:38.179282 21548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:38.180473 21548 hybrid_clock.cc:648] HybridClock initialized: now 1790577518180441 us; error 43 us; skew 500 ppm
Sep 28 06:38:38 dist-test-slave-j0qf krb5kdc[20632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577518, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:38.183714 21548 init.cc:377] Logged in from keytab as kudu/127.5.22.3@KRBTEST.COM (short username kudu)
I20260928 06:38:38.184613 21548 webserver.cc:460] Webserver started at http://127.5.22.3:38741/ using document root <none> and password file <none>
I20260928 06:38:38.185204 21548 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:38.185328 21548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:38.189633 21548 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:38:38.191865 21564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:38.193080 21548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:38:38.193207 21548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "70347f0cca4f47d495db897103efcebb"
format_stamp: "Formatted at 2026-09-28 06:38:17 on dist-test-slave-j0qf"
server_key: "dd8417aa4784926e713612d360f64b4a"
server_key_iv: "6f23a2f84725c56e63a35f6189a99da3"
server_key_version: "encryptionkey@0"
I20260928 06:38:38.193877 21548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:38.206548 21548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:38.209779 21548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:38.209961 21548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:38.210572 21548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:38.211366 21548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:38:38.211433 21548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:38.211498 21548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:38:38.211541 21548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:38.220605 21548 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:46097
I20260928 06:38:38.220676 21677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:46097 every 8 connection(s)
I20260928 06:38:38.221648 21548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:38:38.227560 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 21548
Sep 28 06:38:38 dist-test-slave-j0qf krb5kdc[20632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577518, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:38:38.236737 21409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:38.223476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:35681 (local address 127.5.22.62:43541)
0928 06:38:38.223639 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:38.223644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:38.224699 (+ 1055us) server_negotiation.cc:407] Connection header received
0928 06:38:38.226139 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:38.226142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:38.226190 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:38.226402 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:38.227571 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.228243 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.229474 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.229806 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.229884 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:38.232769 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:38.232774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:38.232779 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:38.232812 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:38.234440 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:38.235045 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:38.235047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:38.235049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:38.235131 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:38.235505 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:38.235510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:38.235512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:38.235725 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:38.235799 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:38.236425 (+ 626us) server_negotiation.cc:299] Negotiation successful
0928 06:38:38.236571 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 06:38:38.236922 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:38.228252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45476 (local address 127.5.22.62:43541)
0928 06:38:38.228506 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:38.228510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:38.228693 (+ 183us) server_negotiation.cc:407] Connection header received
0928 06:38:38.228853 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:38.228856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:38.228897 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:38.228981 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:38.229740 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.230537 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.231396 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.231751 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.231871 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:38.233341 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:38.233344 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:38.233346 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:38.233378 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:38.235199 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:38.235636 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:38.235639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:38.235640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:38.235728 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:38.236026 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:38.236030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:38.236032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:38.236228 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:38.236443 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:38.236647 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 06:38:38.236780 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":65,"threads_started":1}
I20260928 06:38:38.237854 21681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:38.223871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.3:35681)
0928 06:38:38.224506 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:38.224572 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:38.225945 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:38.226556 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:38.226575 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:38.226937 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:38.227421 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:38.227434 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.228428 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.228433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:38.229359 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:38.229369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.229976 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.229981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:38.230171 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:38.230836 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:38.230852 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:38.232605 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:38.234552 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:38.234562 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:38.234579 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:38.234944 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:38.235229 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:38.235235 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:38.235240 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:38.235407 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:38.235851 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:38.235858 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:38.236148 (+ 290us) client_negotiation.cc:769] Sending connection context
0928 06:38:38.236371 (+ 223us) client_negotiation.cc:240] Negotiation successful
0928 06:38:38.236736 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":108,"threads_started":1}
I20260928 06:38:38.238960 21678 heartbeater.cc:347] Connected to a master server at 127.5.22.62:43541
I20260928 06:38:38.239182 21678 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:38.239645 21678 heartbeater.cc:511] Master 127.5.22.62:43541 requested a full tablet report, sending...
I20260928 06:38:38.240602 21215 ts_manager.cc:194] Registered new tserver with Master: 70347f0cca4f47d495db897103efcebb (127.5.22.3:46097)
I20260928 06:38:38.241145 21215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.3@KRBTEST.COM'} at 127.5.22.3:35681
I20260928 06:38:38.250712 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:38:38.254777 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:38.251389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45486 (local address 127.5.22.62:43541)
0928 06:38:38.251548 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:38.251552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:38.251657 (+ 105us) server_negotiation.cc:407] Connection header received
0928 06:38:38.251797 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:38.251799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:38.251898 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:38.251981 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:38:38.252185 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.252773 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.253612 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.253858 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.253939 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:38.254188 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:38:38.254429 (+ 241us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 06:38:38.254643 (+ 214us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45486: token expired
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 06:38:38.254854 21682 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45486: token expired
W20260928 06:38:38.255159 21144 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.5.22.62:43541)
I20260928 06:38:38.262405 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:38.255543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45500 (local address 127.5.22.62:43541)
0928 06:38:38.255673 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:38.255677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:38.255691 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:38:38.255763 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:38.255766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:38.255804 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:38.255891 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:38.256483 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.256963 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.257727 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.257948 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.258048 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:38.259397 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:38.259400 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:38.259402 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:38.259424 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:38.260730 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:38.261154 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:38.261157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:38.261159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:38.261240 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:38.261515 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:38.261519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:38.261521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:38.261724 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:38.261863 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:38.262096 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 06:38:38.262210 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:38:38.264890 21215 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:38:38.268180 21215 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:38:38.276109 21340 tablet_service.cc:1467] Processing CreateTablet for tablet 40b565bf318d495ea318e9094017789d (DEFAULT_TABLE table=test-table [id=185b75118d684d319d44ec7b7b302af0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:38:38.276829 21340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40b565bf318d495ea318e9094017789d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:38.279516 21297 tablet_bootstrap.cc:492] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55: Bootstrap starting.
I20260928 06:38:38.280249 21413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:38.274983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:46097 (local address 127.0.0.1:33788)
0928 06:38:38.275111 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:38.275130 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:38.275212 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:38.276056 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:38.276060 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:38.276075 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:38.276182 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:38.276190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.277514 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.277517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:38.278675 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:38.278686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.279837 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.279841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:38.279949 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:38.279978 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 06:38:38.280026 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 06:38:38.280125 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 06:38:38.280434 21297 tablet_bootstrap.cc:654] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:38.280887 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:38.275206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33788 (local address 127.5.22.3:46097)
0928 06:38:38.275551 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:38.275562 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:38.275585 (+ 23us) server_negotiation.cc:407] Connection header received
0928 06:38:38.275685 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:38.275690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:38.275918 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:38.276082 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:38.276270 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.277386 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.278813 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.279700 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.280513 (+ 813us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:38.280637 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:38.280679 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 06:38:38.280750 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":164,"threads_started":1}
I20260928 06:38:38.281123 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:38.275319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:36871 (local address 127.0.0.1:55718)
0928 06:38:38.275780 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:38.275794 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:38.275892 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:38.276859 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:38.276861 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:38.276875 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:38.276981 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:38.276989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.278265 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.278268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:38.279181 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:38.279189 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.280672 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.280680 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:38.280785 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:38.280808 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 06:38:38.280897 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:38:38.280991 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":119,"threads_started":1}
I20260928 06:38:38.281306 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:38.275760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55718 (local address 127.5.22.2:36871)
0928 06:38:38.276178 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:38.276189 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:38.276228 (+ 39us) server_negotiation.cc:407] Connection header received
0928 06:38:38.276310 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:38.276318 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:38.276549 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:38.276707 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:38.277092 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.278116 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.279298 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.280331 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.280443 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:38.280597 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:38.280954 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 06:38:38.281061 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":108,"threads_started":1}
I20260928 06:38:38.282052 21297 tablet_bootstrap.cc:492] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55: No bootstrap required, opened a new log
I20260928 06:38:38.282156 21297 ts_tablet_manager.cc:1397] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 06:38:38.282913 21297 raft_consensus.cc:348] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } } peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } }
I20260928 06:38:38.283046 21297 raft_consensus.cc:374] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:38.283083 21297 raft_consensus.cc:729] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f779950ca746809a2f124299de1c55, State: Initialized, Role: FOLLOWER
I20260928 06:38:38.283222 21297 consensus_queue.cc:260] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } } peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } }
I20260928 06:38:38.283478 21297 ts_tablet_manager.cc:1428] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 06:38:38.283842 21612 tablet_service.cc:1467] Processing CreateTablet for tablet 40b565bf318d495ea318e9094017789d (DEFAULT_TABLE table=test-table [id=185b75118d684d319d44ec7b7b302af0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:38:38.284166 21479 tablet_service.cc:1467] Processing CreateTablet for tablet 40b565bf318d495ea318e9094017789d (DEFAULT_TABLE table=test-table [id=185b75118d684d319d44ec7b7b302af0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:38:38.284983 21612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40b565bf318d495ea318e9094017789d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:38.285146 21479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40b565bf318d495ea318e9094017789d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:38.291508 21693 tablet_bootstrap.cc:492] T 40b565bf318d495ea318e9094017789d P bd9107d3a54a4f998035e27180f4b8bc: Bootstrap starting.
I20260928 06:38:38.291509 21692 tablet_bootstrap.cc:492] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb: Bootstrap starting.
I20260928 06:38:38.293972 21692 tablet_bootstrap.cc:654] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:38.294149 21693 tablet_bootstrap.cc:654] T 40b565bf318d495ea318e9094017789d P bd9107d3a54a4f998035e27180f4b8bc: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:38.294796 21692 log.cc:824] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:38.294935 21693 log.cc:824] T 40b565bf318d495ea318e9094017789d P bd9107d3a54a4f998035e27180f4b8bc: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:38.296762 21692 tablet_bootstrap.cc:492] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb: No bootstrap required, opened a new log
I20260928 06:38:38.296793 21693 tablet_bootstrap.cc:492] T 40b565bf318d495ea318e9094017789d P bd9107d3a54a4f998035e27180f4b8bc: No bootstrap required, opened a new log
I20260928 06:38:38.296967 21692 ts_tablet_manager.cc:1397] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 06:38:38.296965 21693 ts_tablet_manager.cc:1397] T 40b565bf318d495ea318e9094017789d P bd9107d3a54a4f998035e27180f4b8bc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 06:38:38.300057 21693 raft_consensus.cc:348] T 40b565bf318d495ea318e9094017789d P bd9107d3a54a4f998035e27180f4b8bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } } peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } }
I20260928 06:38:38.300052 21692 raft_consensus.cc:348] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } } peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } }
I20260928 06:38:38.300287 21692 raft_consensus.cc:374] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:38.300292 21693 raft_consensus.cc:374] T 40b565bf318d495ea318e9094017789d P bd9107d3a54a4f998035e27180f4b8bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:38.300321 21692 raft_consensus.cc:729] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70347f0cca4f47d495db897103efcebb, State: Initialized, Role: FOLLOWER
I20260928 06:38:38.300341 21693 raft_consensus.cc:729] T 40b565bf318d495ea318e9094017789d P bd9107d3a54a4f998035e27180f4b8bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd9107d3a54a4f998035e27180f4b8bc, State: Initialized, Role: FOLLOWER
I20260928 06:38:38.300745 21692 consensus_queue.cc:260] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } } peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } }
I20260928 06:38:38.300745 21693 consensus_queue.cc:260] T 40b565bf318d495ea318e9094017789d P bd9107d3a54a4f998035e27180f4b8bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } } peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } }
I20260928 06:38:38.301417 21678 heartbeater.cc:503] Master 127.5.22.62:43541 was elected leader, sending a full tablet report...
I20260928 06:38:38.301748 21692 ts_tablet_manager.cc:1428] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260928 06:38:38.301759 21693 ts_tablet_manager.cc:1428] T 40b565bf318d495ea318e9094017789d P bd9107d3a54a4f998035e27180f4b8bc: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260928 06:38:38.302050 21546 heartbeater.cc:503] Master 127.5.22.62:43541 was elected leader, sending a full tablet report...
W20260928 06:38:38.333527 21408 tablet.cc:2369] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:38:38.472970 21680 tablet.cc:2369] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:38:38.539522 21547 tablet.cc:2369] T 40b565bf318d495ea318e9094017789d P bd9107d3a54a4f998035e27180f4b8bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:38:38.553678 21696 raft_consensus.cc:482] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:38:38.553898 21696 raft_consensus.cc:504] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } } peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } }
I20260928 06:38:38.555100 21696 leader_election.cc:288] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd9107d3a54a4f998035e27180f4b8bc (127.5.22.2:36871), 76f779950ca746809a2f124299de1c55 (127.5.22.1:34999)
I20260928 06:38:38.560160 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:38.555467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:47619 (local address 127.5.22.2:36871)
0928 06:38:38.555619 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:38.555626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:38.555649 (+ 23us) server_negotiation.cc:407] Connection header received
0928 06:38:38.555711 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:38.555716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:38.555811 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:38.555908 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:38.556227 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.557074 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.558812 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.559459 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.559578 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:38.559645 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:38.559954 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 06:38:38.560031 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 06:38:38.560163 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:38.555668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:46003 (local address 127.5.22.1:34999)
0928 06:38:38.555944 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:38.555948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:38.555962 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:38:38.556088 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:38.556091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:38.556132 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:38.556233 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:38.556465 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.557189 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.558820 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.559469 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.559578 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:38.559647 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:38.559954 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 06:38:38.560032 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":81,"threads_started":1}
I20260928 06:38:38.560164 21681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:38.555346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:36871 (local address 127.5.22.3:47619)
0928 06:38:38.555485 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:38.555504 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:38.555607 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:38.555944 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:38.555950 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:38.555973 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:38.556114 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:38.556119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.557187 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.557190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:38.558673 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:38.558684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.559593 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.559597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:38.559734 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:38.559833 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 06:38:38.559916 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 06:38:38.559993 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":189}
I20260928 06:38:38.560173 21698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:38.555478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:34999 (local address 127.5.22.3:46003)
0928 06:38:38.555891 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:38.555907 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:38.555981 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:38.556255 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:38.556257 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:38.556272 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:38.556357 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:38.556362 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.557287 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.557290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:38.558673 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:38.558684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.559593 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.559597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:38.559732 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:38.559838 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 06:38:38.559916 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:38:38.559993 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":96,"thread_start_us":127,"threads_started":1}
I20260928 06:38:38.560988 21360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40b565bf318d495ea318e9094017789d" candidate_uuid: "70347f0cca4f47d495db897103efcebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76f779950ca746809a2f124299de1c55" is_pre_election: true
I20260928 06:38:38.561034 21499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40b565bf318d495ea318e9094017789d" candidate_uuid: "70347f0cca4f47d495db897103efcebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd9107d3a54a4f998035e27180f4b8bc" is_pre_election: true
I20260928 06:38:38.561303 21360 raft_consensus.cc:2454] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70347f0cca4f47d495db897103efcebb in term 0.
I20260928 06:38:38.561301 21499 raft_consensus.cc:2454] T 40b565bf318d495ea318e9094017789d P bd9107d3a54a4f998035e27180f4b8bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70347f0cca4f47d495db897103efcebb in term 0.
I20260928 06:38:38.561743 21568 leader_election.cc:302] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70347f0cca4f47d495db897103efcebb, 76f779950ca746809a2f124299de1c55; no voters:
I20260928 06:38:38.562043 21696 raft_consensus.cc:2781] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:38:38.562144 21696 raft_consensus.cc:482] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:38:38.562203 21696 raft_consensus.cc:3037] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:38.563678 21696 raft_consensus.cc:504] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } } peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } }
I20260928 06:38:38.564095 21696 leader_election.cc:288] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb [CANDIDATE]: Term 1 election: Requested vote from peers bd9107d3a54a4f998035e27180f4b8bc (127.5.22.2:36871), 76f779950ca746809a2f124299de1c55 (127.5.22.1:34999)
I20260928 06:38:38.564497 21499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40b565bf318d495ea318e9094017789d" candidate_uuid: "70347f0cca4f47d495db897103efcebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd9107d3a54a4f998035e27180f4b8bc"
I20260928 06:38:38.564584 21360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40b565bf318d495ea318e9094017789d" candidate_uuid: "70347f0cca4f47d495db897103efcebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76f779950ca746809a2f124299de1c55"
I20260928 06:38:38.564641 21499 raft_consensus.cc:3037] T 40b565bf318d495ea318e9094017789d P bd9107d3a54a4f998035e27180f4b8bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:38.564720 21360 raft_consensus.cc:3037] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:38.566082 21360 raft_consensus.cc:2454] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70347f0cca4f47d495db897103efcebb in term 1.
I20260928 06:38:38.566098 21499 raft_consensus.cc:2454] T 40b565bf318d495ea318e9094017789d P bd9107d3a54a4f998035e27180f4b8bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70347f0cca4f47d495db897103efcebb in term 1.
I20260928 06:38:38.566414 21565 leader_election.cc:302] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70347f0cca4f47d495db897103efcebb, bd9107d3a54a4f998035e27180f4b8bc; no voters:
I20260928 06:38:38.566726 21696 raft_consensus.cc:2781] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:38:38.567003 21696 raft_consensus.cc:686] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb [term 1 LEADER]: Becoming Leader. State: Replica: 70347f0cca4f47d495db897103efcebb, State: Running, Role: LEADER
I20260928 06:38:38.567322 21696 consensus_queue.cc:237] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } } peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } }
I20260928 06:38:38.570632 21214 catalog_manager.cc:5495] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb reported cstate change: term changed from 0 to 1, leader changed from <none> to 70347f0cca4f47d495db897103efcebb (127.5.22.3). New cstate: current_term: 1 leader_uuid: "70347f0cca4f47d495db897103efcebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } health_report { overall_health: UNKNOWN } } }
I20260928 06:38:38.584321 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:38.580580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33790 (local address 127.5.22.3:46097)
0928 06:38:38.580728 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:38.580732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:38.580747 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:38:38.580914 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:38.580917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:38.580972 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:38.581079 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:38:38.581311 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.581831 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.582797 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:38.583016 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:38.583124 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:38.583422 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:38:38.583903 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:38:38.584045 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:38.584105 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 06:38:38.584161 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:38:38.590197 21360 raft_consensus.cc:1261] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [term 1 FOLLOWER]: Refusing update from remote peer 70347f0cca4f47d495db897103efcebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:38:38.590201 21499 raft_consensus.cc:1261] T 40b565bf318d495ea318e9094017789d P bd9107d3a54a4f998035e27180f4b8bc [term 1 FOLLOWER]: Refusing update from remote peer 70347f0cca4f47d495db897103efcebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:38:38.591051 21696 consensus_queue.cc:1035] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:38:38.591355 21700 consensus_queue.cc:1035] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:38:38.598156 21704 mvcc.cc:205] Tried to move back new op lower bound from 7334205516138110976 to 7334205516054065152. Current Snapshot: MvccSnapshot[applied={T|T < 7334205516138110976}]
I20260928 06:38:38.599496 21705 mvcc.cc:205] Tried to move back new op lower bound from 7334205516138110976 to 7334205516054065152. Current Snapshot: MvccSnapshot[applied={T|T < 7334205516138110976}]
I20260928 06:38:38.606781 21215 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:38:38.606938 21215 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:38:38.609222 21215 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194: Sending DeleteTablet for 3 replicas of tablet 40b565bf318d495ea318e9094017789d
I20260928 06:38:38.609959 21340 tablet_service.cc:1514] Processing DeleteTablet for tablet 40b565bf318d495ea318e9094017789d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:38:38 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:52992
I20260928 06:38:38.609978 21612 tablet_service.cc:1514] Processing DeleteTablet for tablet 40b565bf318d495ea318e9094017789d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:38:38 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:33788
I20260928 06:38:38.610118 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 21255
I20260928 06:38:38.611081 21479 tablet_service.cc:1514] Processing DeleteTablet for tablet 40b565bf318d495ea318e9094017789d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:38:38 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:55718
I20260928 06:38:38.611560 21714 tablet_replica.cc:330] T 40b565bf318d495ea318e9094017789d P bd9107d3a54a4f998035e27180f4b8bc: stopping tablet replica
I20260928 06:38:38.611562 21713 tablet_replica.cc:330] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb: stopping tablet replica
I20260928 06:38:38.611871 21713 raft_consensus.cc:2229] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:38:38.611871 21714 raft_consensus.cc:2229] T 40b565bf318d495ea318e9094017789d P bd9107d3a54a4f998035e27180f4b8bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:38:38.612206 21714 raft_consensus.cc:2258] T 40b565bf318d495ea318e9094017789d P bd9107d3a54a4f998035e27180f4b8bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:38:38.612251 21713 raft_consensus.cc:2258] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:38:38.613796 21714 ts_tablet_manager.cc:1905] T 40b565bf318d495ea318e9094017789d P bd9107d3a54a4f998035e27180f4b8bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:38:38.614624 21713 ts_tablet_manager.cc:1905] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:38:38.616891 21713 ts_tablet_manager.cc:1918] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:38:38.617013 21713 log.cc:1196] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/40b565bf318d495ea318e9094017789d
I20260928 06:38:38.617424 21713 ts_tablet_manager.cc:1939] T 40b565bf318d495ea318e9094017789d P 70347f0cca4f47d495db897103efcebb: Deleting consensus metadata
I20260928 06:38:38.617425 21714 ts_tablet_manager.cc:1918] T 40b565bf318d495ea318e9094017789d P bd9107d3a54a4f998035e27180f4b8bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:38:38.617529 21714 log.cc:1196] T 40b565bf318d495ea318e9094017789d P bd9107d3a54a4f998035e27180f4b8bc: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/wal/wals/40b565bf318d495ea318e9094017789d
I20260928 06:38:38.618036 21714 ts_tablet_manager.cc:1939] T 40b565bf318d495ea318e9094017789d P bd9107d3a54a4f998035e27180f4b8bc: Deleting consensus metadata
I20260928 06:38:38.618853 21202 catalog_manager.cc:4841] TS 70347f0cca4f47d495db897103efcebb (127.5.22.3:46097): tablet 40b565bf318d495ea318e9094017789d (table test-table [id=185b75118d684d319d44ec7b7b302af0]) successfully deleted
W20260928 06:38:38.619235 21202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.22.1:34999 (error 108)
I20260928 06:38:38.619424 21199 catalog_manager.cc:4841] TS bd9107d3a54a4f998035e27180f4b8bc (127.5.22.2:36871): tablet 40b565bf318d495ea318e9094017789d (table test-table [id=185b75118d684d319d44ec7b7b302af0]) successfully deleted
I20260928 06:38:38.619529 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 21411
I20260928 06:38:38.620867 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:38.620544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:34999 (local address 127.0.0.1:42200)
0928 06:38:38.620681 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:38.620782 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.1:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 06:38:38.621093 21202 catalog_manager.cc:4584] TS 76f779950ca746809a2f124299de1c55 (127.5.22.1:34999): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 40b565bf318d495ea318e9094017789d: Network error: Client connection negotiation failed: client connection to 127.5.22.1:34999: connect: Connection refused (error 111)
I20260928 06:38:38.628125 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 21548
I20260928 06:38:38.635236 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 21182
I20260928 06:38:58.642143 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:43541
--webserver_interface=127.5.22.62
--webserver_port=41199
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:43541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:38:58.754946 21716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:58.755262 21716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:58.755339 21716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:58.759346 21716 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 06:38:58.759429 21716 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 06:38:58.759457 21716 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 06:38:58.759490 21716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:38:58.759531 21716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:58.759573 21716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:58.759610 21716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:38:58.759646 21716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:38:58.764935 21716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:43541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:43541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=41199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:38:58.766089 21716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:58.767210 21716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:58.776382 21721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:58.776542 21716 server_base.cc:1034] running on GCE node
W20260928 06:38:58.776643 21724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:58.776700 21722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:58.777264 21716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:58.778594 21716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:58.779800 21716 hybrid_clock.cc:648] HybridClock initialized: now 1790577538779772 us; error 42 us; skew 500 ppm
Sep 28 06:38:58 dist-test-slave-j0qf krb5kdc[20632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577538, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:58.782936 21716 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:38:58.783855 21716 webserver.cc:460] Webserver started at http://127.5.22.62:41199/ using document root <none> and password file <none>
I20260928 06:38:58.784433 21716 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:58.784498 21716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:58.788633 21716 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:38:58.790941 21731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:58.792209 21716 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:38:58.792352 21716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "a58cb30fef744b048351d0c17588c194"
format_stamp: "Formatted at 2026-09-28 06:38:16 on dist-test-slave-j0qf"
server_key: "921bad6934aa05c44df053c02f840b14"
server_key_iv: "afb617f63704388a57903759b7fea2ea"
server_key_version: "encryptionkey@0"
I20260928 06:38:58.792927 21716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:58.816223 21716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:58.819677 21716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:58.819908 21716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:58.827143 21716 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:43541
I20260928 06:38:58.827483 21783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:43541 every 8 connection(s)
I20260928 06:38:58.828323 21716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:38:58.829041 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 21716
I20260928 06:38:58.829547 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:34999
--local_ip_for_outbound_sockets=127.5.22.1
--tserver_master_addrs=127.5.22.62:43541
--webserver_port=40357
--webserver_interface=127.5.22.1
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:38:58.833732 21784 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:38:58.835494 21784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194: Bootstrap starting.
I20260928 06:38:58.853885 21784 log.cc:824] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:58.865264 21784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:38:58.865670 21784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194: Bootstrap complete.
I20260928 06:38:58.869906 21784 raft_consensus.cc:348] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58cb30fef744b048351d0c17588c194" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43541 } }
I20260928 06:38:58.870311 21784 raft_consensus.cc:729] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a58cb30fef744b048351d0c17588c194, State: Initialized, Role: FOLLOWER
I20260928 06:38:58.870913 21784 consensus_queue.cc:260] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58cb30fef744b048351d0c17588c194" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43541 } }
I20260928 06:38:58.871060 21784 raft_consensus.cc:388] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:38:58.871152 21784 raft_consensus.cc:482] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:38:58.871253 21784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [term 2 FOLLOWER]: Advancing to term 3
I20260928 06:38:58.873160 21784 raft_consensus.cc:504] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58cb30fef744b048351d0c17588c194" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43541 } }
I20260928 06:38:58.873509 21784 leader_election.cc:302] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a58cb30fef744b048351d0c17588c194; no voters:
I20260928 06:38:58.873849 21784 leader_election.cc:288] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 06:38:58.874006 21788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 06:38:58.874264 21788 raft_consensus.cc:686] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [term 3 LEADER]: Becoming Leader. State: Replica: a58cb30fef744b048351d0c17588c194, State: Running, Role: LEADER
I20260928 06:38:58.874612 21788 consensus_queue.cc:237] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58cb30fef744b048351d0c17588c194" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43541 } }
I20260928 06:38:58.875139 21784 sys_catalog.cc:564] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:38:58.875607 21788 sys_catalog.cc:455] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a58cb30fef744b048351d0c17588c194. Latest consensus state: current_term: 3 leader_uuid: "a58cb30fef744b048351d0c17588c194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58cb30fef744b048351d0c17588c194" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43541 } } }
I20260928 06:38:58.875725 21788 sys_catalog.cc:458] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [sys.catalog]: This master's current role is: LEADER
I20260928 06:38:58.876334 21789 sys_catalog.cc:455] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a58cb30fef744b048351d0c17588c194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58cb30fef744b048351d0c17588c194" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 43541 } } }
I20260928 06:38:58.876426 21789 sys_catalog.cc:458] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194 [sys.catalog]: This master's current role is: LEADER
I20260928 06:38:58.877276 21792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:38:58.882606 21792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:38:58.883246 21792 catalog_manager.cc:1259] Loaded cluster ID: a93ffe9a51214f6b97f899075cd011ea
I20260928 06:38:58.883319 21792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:38:58.885810 21792 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:38:58.886667 21792 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194: Loaded TSK: 0
I20260928 06:38:58.886755 21792 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a58cb30fef744b048351d0c17588c194: Loaded TSK: 1
F20260928 06:38:58.887477 21792 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff4b208c9ad google::LogMessage::Fail() at ??:0
@ 0x7ff4b208f88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff4b208c4de google::LogMessage::Flush() at ??:0
@ 0x7ff4b208da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff4b4d5018d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff4b4d48f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff4bc9f17bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff4bc9cb99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff4bc9cc178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7ff4bca0ddf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7ff4bca2c82b std::function<>::operator()() at ??:0
@ 0x7ff4bc9cbb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7ff4bc9ccf3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff4bc9c9579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7ff4bca0d640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff4b42de3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff4b42dec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7ff4b42e03f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff4b42cf512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff4b59136db start_thread at ??:0
@ 0x7ff4b0b8a61f clone at ??:0
W20260928 06:38:58.989676 21786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:58.990043 21786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:58.990131 21786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:58.999461 21786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:58.999540 21786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:58.999668 21786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:38:59.010679 21786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:34999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=40357
--tserver_master_addrs=127.5.22.62:43541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:38:59.012235 21786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:59.013423 21786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:59.023101 21835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:59.023132 21832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:59.023133 21833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:59.023487 21786 server_base.cc:1034] running on GCE node
I20260928 06:38:59.023900 21786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:59.024628 21786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:59.025859 21786 hybrid_clock.cc:648] HybridClock initialized: now 1790577539025823 us; error 52 us; skew 500 ppm
Sep 28 06:38:59 dist-test-slave-j0qf krb5kdc[20632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577539, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:59.028941 21786 init.cc:377] Logged in from keytab as kudu/127.5.22.1@KRBTEST.COM (short username kudu)
I20260928 06:38:59.029875 21786 webserver.cc:460] Webserver started at http://127.5.22.1:40357/ using document root <none> and password file <none>
I20260928 06:38:59.030520 21786 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:59.030586 21786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:59.034778 21786 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:38:59.037104 21842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:59.038684 21786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:38:59.038826 21786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "76f779950ca746809a2f124299de1c55"
format_stamp: "Formatted at 2026-09-28 06:38:16 on dist-test-slave-j0qf"
server_key: "fab94183b6424fa54221b652212d737c"
server_key_iv: "1287890ecc8c241a725790426808791a"
server_key_version: "encryptionkey@0"
I20260928 06:38:59.039489 21786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:59.060667 21786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:59.064167 21786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:59.064481 21786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:59.065155 21786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:59.066287 21849 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:38:59.070801 21786 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:38:59.070889 21786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260928 06:38:59.070971 21786 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:38:59.072697 21786 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:38:59.072757 21786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 06:38:59.073017 21849 tablet_bootstrap.cc:492] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55: Bootstrap starting.
I20260928 06:38:59.083877 21786 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:34999
I20260928 06:38:59.084065 21956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:34999 every 8 connection(s)
I20260928 06:38:59.085373 21786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:38:59.087875 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 21786
I20260928 06:38:59.088368 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:36871
--local_ip_for_outbound_sockets=127.5.22.2
--tserver_master_addrs=127.5.22.62:43541
--webserver_port=33651
--webserver_interface=127.5.22.2
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:38:59.094877 21849 log.cc:824] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:59.105206 21849 tablet_bootstrap.cc:492] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:38:59.105604 21849 tablet_bootstrap.cc:492] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55: Bootstrap complete.
I20260928 06:38:59.107105 21849 ts_tablet_manager.cc:1397] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55: Time spent bootstrapping tablet: real 0.034s user 0.023s sys 0.010s
I20260928 06:38:59.110172 21849 raft_consensus.cc:348] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } } peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } }
I20260928 06:38:59.110718 21849 raft_consensus.cc:729] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f779950ca746809a2f124299de1c55, State: Initialized, Role: FOLLOWER
I20260928 06:38:59.111222 21849 consensus_queue.cc:260] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } } peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } }
I20260928 06:38:59.112316 21849 ts_tablet_manager.cc:1428] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260928 06:38:59.132630 21959 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:38:59.087852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.1:53721)
0928 06:38:59.090624 (+ 2772us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:59.090693 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:59.091822 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:59.131888 (+ 40066us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2517,"thread_start_us":174,"threads_started":1}
W20260928 06:38:59.133358 21957 heartbeater.cc:644] Failed to heartbeat to 127.5.22.62:43541 (0 consecutive failures): Network error: Failed to ping master at 127.5.22.62:43541: Client connection negotiation failed: client connection to 127.5.22.62:43541: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:38:59.134089 21959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:59.133833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.1:51861)
0928 06:38:59.133961 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:59.134034 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 06:38:59.134927 21959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:59.134676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.1:35263)
0928 06:38:59.134802 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:59.134857 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 06:38:59.135138 21957 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:38:59.209180 21960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:59.209420 21960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:59.209462 21960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:59.213541 21960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:59.213598 21960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:59.213686 21960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:38:59.218948 21960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:36871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=33651
--tserver_master_addrs=127.5.22.62:43541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:38:59.220011 21960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:59.221048 21960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:59.230598 21970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:59.230600 21967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:59.230598 21968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:59.230845 21960 server_base.cc:1034] running on GCE node
I20260928 06:38:59.231256 21960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:59.231879 21960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:59.233055 21960 hybrid_clock.cc:648] HybridClock initialized: now 1790577539233011 us; error 56 us; skew 500 ppm
Sep 28 06:38:59 dist-test-slave-j0qf krb5kdc[20632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577539, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:59.235924 21960 init.cc:377] Logged in from keytab as kudu/127.5.22.2@KRBTEST.COM (short username kudu)
I20260928 06:38:59.236944 21960 webserver.cc:460] Webserver started at http://127.5.22.2:33651/ using document root <none> and password file <none>
I20260928 06:38:59.237548 21960 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:59.237614 21960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:59.241819 21960 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:38:59.244329 21977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:59.245690 21960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:38:59.245839 21960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "bd9107d3a54a4f998035e27180f4b8bc"
format_stamp: "Formatted at 2026-09-28 06:38:16 on dist-test-slave-j0qf"
server_key: "15044b77039649d331b6173c514aa563"
server_key_iv: "f3ef15d21f657fd8ba3d3f3b500291dc"
server_key_version: "encryptionkey@0"
I20260928 06:38:59.246481 21960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:59.270781 21960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:59.274058 21960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:59.274274 21960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:59.274959 21960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:59.275839 21960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:38:59.275887 21960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:59.275957 21960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:38:59.276001 21960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:59.285319 21960 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:36871
I20260928 06:38:59.285341 22092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:36871 every 8 connection(s)
I20260928 06:38:59.286352 21960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:38:59.289698 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 21960
I20260928 06:38:59.290123 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:46097
--local_ip_for_outbound_sockets=127.5.22.3
--tserver_master_addrs=127.5.22.62:43541
--webserver_port=38741
--webserver_interface=127.5.22.3
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:38:59.290273 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:59.288465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.2:34059)
0928 06:38:59.288947 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:59.289438 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":116,"threads_started":1}
W20260928 06:38:59.290764 22093 heartbeater.cc:644] Failed to heartbeat to 127.5.22.62:43541 (0 consecutive failures): Network error: Failed to ping master at 127.5.22.62:43541: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
I20260928 06:38:59.291422 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:59.291198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.2:60121)
0928 06:38:59.291324 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:59.291372 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 06:38:59.292162 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:59.291897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.2:51825)
0928 06:38:59.292026 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:59.292088 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 06:38:59.292424 22093 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:38:59.401616 22096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:59.401868 22096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:59.401912 22096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:59.405834 22096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:59.405895 22096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:59.405953 22096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:38:59.411037 22096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:46097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=38741
--tserver_master_addrs=127.5.22.62:43541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:38:59.411983 22096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:59.412933 22096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:59.422551 22102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:59.422551 22103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:59.422530 22105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:59.422530 22096 server_base.cc:1034] running on GCE node
I20260928 06:38:59.423185 22096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:59.423877 22096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:59.425047 22096 hybrid_clock.cc:648] HybridClock initialized: now 1790577539425033 us; error 26 us; skew 500 ppm
Sep 28 06:38:59 dist-test-slave-j0qf krb5kdc[20632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577539, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:59.427996 22096 init.cc:377] Logged in from keytab as kudu/127.5.22.3@KRBTEST.COM (short username kudu)
I20260928 06:38:59.428939 22096 webserver.cc:460] Webserver started at http://127.5.22.3:38741/ using document root <none> and password file <none>
I20260928 06:38:59.429515 22096 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:59.429580 22096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:59.433655 22096 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:38:59.435936 22112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:59.437314 22096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:38:59.437453 22096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "70347f0cca4f47d495db897103efcebb"
format_stamp: "Formatted at 2026-09-28 06:38:17 on dist-test-slave-j0qf"
server_key: "dd8417aa4784926e713612d360f64b4a"
server_key_iv: "6f23a2f84725c56e63a35f6189a99da3"
server_key_version: "encryptionkey@0"
I20260928 06:38:59.438001 22096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:59.466094 22096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:59.469405 22096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:59.469620 22096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:59.470244 22096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:59.471259 22096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:38:59.471314 22096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:59.471379 22096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:38:59.471413 22096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:59.480736 22096 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:46097
I20260928 06:38:59.480760 22226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:46097 every 8 connection(s)
I20260928 06:38:59.481834 22096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:38:59.485718 22229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:59.484043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.3:43853)
0928 06:38:59.484514 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:59.484999 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":115,"threads_started":1}
W20260928 06:38:59.486244 22227 heartbeater.cc:644] Failed to heartbeat to 127.5.22.62:43541 (0 consecutive failures): Network error: Failed to ping master at 127.5.22.62:43541: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
I20260928 06:38:59.486904 22229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:59.486677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.3:44795)
0928 06:38:59.486791 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:59.486843 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 06:38:59.487556 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 22096
I20260928 06:38:59.487709 22229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:59.487478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.3:56897)
0928 06:38:59.487582 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:59.487648 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 06:38:59.487895 22227 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:39:00.136217 22231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:00.135771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.1:59917)
0928 06:39:00.136046 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:00.136105 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":89,"threads_started":1}
I20260928 06:39:00.293486 22232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:00.293032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.2:57255)
0928 06:39:00.293304 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:00.293377 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":73,"threads_started":1}
I20260928 06:39:00.488832 22233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:00.488447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.3:49551)
0928 06:39:00.488679 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:00.488733 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":65,"threads_started":1}
I20260928 06:39:00.571975 22234 raft_consensus.cc:482] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:39:00.572242 22234 raft_consensus.cc:504] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } } peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } }
I20260928 06:39:00.573329 22234 leader_election.cc:288] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 70347f0cca4f47d495db897103efcebb (127.5.22.3:46097), bd9107d3a54a4f998035e27180f4b8bc (127.5.22.2:36871)
Sep 28 06:39:00 dist-test-slave-j0qf krb5kdc[20632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577539, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for kudu/127.5.22.3@KRBTEST.COM
Sep 28 06:39:00 dist-test-slave-j0qf krb5kdc[20632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577539, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for kudu/127.5.22.2@KRBTEST.COM
I20260928 06:39:00.587952 22236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:00.573879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:43173 (local address 127.5.22.3:46097)
0928 06:39:00.574218 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:00.574244 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:00.574280 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:39:00.575095 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:00.575101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:00.575435 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:00.575739 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:00.576992 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:00.577971 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:00.579426 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:00.579907 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:00.580068 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:00.582878 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:00.582884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:00.582898 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:00.582934 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:00.585293 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:00.585918 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:00.585924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:00.585936 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:00.586047 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:00.586413 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:00.586418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:00.586420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:00.586815 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:00.586959 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:00.587492 (+ 533us) server_negotiation.cc:299] Negotiation successful
0928 06:39:00.587758 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":85,"threads_started":1}
I20260928 06:39:00.587934 22235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:00.573626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:46097 (local address 127.5.22.1:43173)
0928 06:39:00.574081 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:00.574095 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:00.574195 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:00.575787 (+ 1592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:00.575792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:00.576333 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:00.576822 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:00.576836 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:00.578118 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:00.578121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:00.579211 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:00.579221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:00.580038 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:00.580042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:00.580151 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:00.580877 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:00.580893 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:00.582611 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:00.585456 (+ 2845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:00.585458 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:00.585460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:00.585791 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:00.586144 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:00.586149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:00.586151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:00.586290 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:00.586935 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:00.586941 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:00.587181 (+ 240us) client_negotiation.cc:769] Sending connection context
0928 06:39:00.587361 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 06:39:00.587735 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":276,"spinlock_wait_cycles":7680,"thread_start_us":65,"threads_started":1}
I20260928 06:39:00.588013 22237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:00.573847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:44345 (local address 127.5.22.2:36871)
0928 06:39:00.574362 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:00.574386 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:00.574422 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:39:00.575113 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:00.575118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:00.575435 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:00.575739 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:00.576994 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:00.577981 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:00.579330 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:00.579838 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:00.579959 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:00.583177 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:00.583184 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:00.583200 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:00.583236 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:00.585291 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:00.585958 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:00.585963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:00.585967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:00.586079 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:00.586423 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:00.586430 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:00.586432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:00.586815 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:00.586971 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:00.587531 (+ 560us) server_negotiation.cc:299] Negotiation successful
0928 06:39:00.587819 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":135,"threads_started":1}
I20260928 06:39:00.588673 22231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:00.573614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:36871 (local address 127.5.22.1:44345)
0928 06:39:00.573818 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:00.573845 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:00.573949 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:00.575786 (+ 1837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:00.575791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:00.576337 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:00.576821 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:00.576837 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:00.578134 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:00.578137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:00.579210 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:00.579220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:00.579967 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:00.579974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:00.580147 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:00.580894 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:00.580906 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:00.582989 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:00.585420 (+ 2431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:00.585425 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:00.585436 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:00.585806 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:00.586181 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:00.586187 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:00.586192 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:00.586300 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:00.586939 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:00.586942 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:00.587182 (+ 240us) client_negotiation.cc:769] Sending connection context
0928 06:39:00.587369 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 06:39:00.587825 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":274,"spinlock_wait_cycles":39040}
W20260928 06:39:00.590626 21846 leader_election.cc:341] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70347f0cca4f47d495db897103efcebb (127.5.22.3:46097): Not found: Tablet not found: 40b565bf318d495ea318e9094017789d
W20260928 06:39:00.590929 21843 leader_election.cc:341] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bd9107d3a54a4f998035e27180f4b8bc (127.5.22.2:36871): Not found: Tablet not found: 40b565bf318d495ea318e9094017789d
I20260928 06:39:00.591033 21843 leader_election.cc:302] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76f779950ca746809a2f124299de1c55; no voters: 70347f0cca4f47d495db897103efcebb, bd9107d3a54a4f998035e27180f4b8bc
I20260928 06:39:00.591384 22234 raft_consensus.cc:2726] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:39:01.137521 22238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:01.137094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.1:58183)
0928 06:39:01.137357 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:01.137420 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":87,"threads_started":1}
I20260928 06:39:01.294793 22239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:01.294295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.2:56395)
0928 06:39:01.294643 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:01.294701 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":80,"threads_started":1}
I20260928 06:39:01.490288 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:01.489712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.3:56615)
0928 06:39:01.490081 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:01.490172 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":112,"threads_started":1}
I20260928 06:39:02.138852 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:02.138308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.1:55803)
0928 06:39:02.138649 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:02.138710 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":77,"threads_started":1}
I20260928 06:39:02.295917 22242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:02.295520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.2:39957)
0928 06:39:02.295757 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:02.295813 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":74,"threads_started":1}
I20260928 06:39:02.336086 22243 raft_consensus.cc:482] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:39:02.336218 22243 raft_consensus.cc:504] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } } peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } }
I20260928 06:39:02.336748 22243 leader_election.cc:288] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 70347f0cca4f47d495db897103efcebb (127.5.22.3:46097), bd9107d3a54a4f998035e27180f4b8bc (127.5.22.2:36871)
W20260928 06:39:02.337669 21843 leader_election.cc:341] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bd9107d3a54a4f998035e27180f4b8bc (127.5.22.2:36871): Not found: Tablet not found: 40b565bf318d495ea318e9094017789d
W20260928 06:39:02.337760 21846 leader_election.cc:341] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70347f0cca4f47d495db897103efcebb (127.5.22.3:46097): Not found: Tablet not found: 40b565bf318d495ea318e9094017789d
I20260928 06:39:02.337886 21846 leader_election.cc:302] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76f779950ca746809a2f124299de1c55; no voters: 70347f0cca4f47d495db897103efcebb, bd9107d3a54a4f998035e27180f4b8bc
I20260928 06:39:02.338095 22243 raft_consensus.cc:2726] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:39:02.491488 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:02.491042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.3:44151)
0928 06:39:02.491329 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:02.491388 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":76,"threads_started":1}
I20260928 06:39:03.140136 22245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:03.139711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.1:51777)
0928 06:39:03.139977 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:03.140039 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":83,"threads_started":1}
I20260928 06:39:03.297235 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:03.296790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.2:47011)
0928 06:39:03.297065 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:03.297132 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":76,"threads_started":1}
I20260928 06:39:03.492854 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:03.492408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.3:40145)
0928 06:39:03.492691 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:03.492755 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":84,"threads_started":1}
I20260928 06:39:04.141927 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:04.141202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.1:33703)
0928 06:39:04.141702 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:04.141804 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":132,"threads_started":1}
I20260928 06:39:04.298599 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:04.298151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.2:39305)
0928 06:39:04.298452 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:04.298505 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":71,"threads_started":1}
I20260928 06:39:04.494489 22250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:04.493701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.3:41303)
0928 06:39:04.494155 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:04.494385 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":87,"threads_started":1}
I20260928 06:39:05.143448 22251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:05.142914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.1:50993)
0928 06:39:05.143233 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:05.143306 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":77,"threads_started":1}
I20260928 06:39:05.255928 22252 raft_consensus.cc:482] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:39:05.256064 22252 raft_consensus.cc:504] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } } peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } }
I20260928 06:39:05.256479 22252 leader_election.cc:288] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 70347f0cca4f47d495db897103efcebb (127.5.22.3:46097), bd9107d3a54a4f998035e27180f4b8bc (127.5.22.2:36871)
W20260928 06:39:05.257263 21846 leader_election.cc:341] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70347f0cca4f47d495db897103efcebb (127.5.22.3:46097): Not found: Tablet not found: 40b565bf318d495ea318e9094017789d
W20260928 06:39:05.257432 21843 leader_election.cc:341] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bd9107d3a54a4f998035e27180f4b8bc (127.5.22.2:36871): Not found: Tablet not found: 40b565bf318d495ea318e9094017789d
I20260928 06:39:05.257556 21843 leader_election.cc:302] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76f779950ca746809a2f124299de1c55; no voters: 70347f0cca4f47d495db897103efcebb, bd9107d3a54a4f998035e27180f4b8bc
I20260928 06:39:05.257714 22252 raft_consensus.cc:2726] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:39:05.299780 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:05.299327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.2:40853)
0928 06:39:05.299610 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:05.299687 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":106,"threads_started":1}
I20260928 06:39:05.495899 22254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:05.495401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.3:58777)
0928 06:39:05.495696 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:05.495758 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":97,"threads_started":1}
I20260928 06:39:06.144655 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:06.144206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.1:42493)
0928 06:39:06.144478 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:06.144550 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":80,"threads_started":1}
I20260928 06:39:06.301148 22256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:06.300647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.2:50541)
0928 06:39:06.301003 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:06.301057 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":94,"threads_started":1}
I20260928 06:39:06.497270 22257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:06.496697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.3:44115)
0928 06:39:06.497033 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:06.497097 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":96,"threads_started":1}
I20260928 06:39:07.145957 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:07.145506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.1:33835)
0928 06:39:07.145789 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:07.145856 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":93,"threads_started":1}
I20260928 06:39:07.302575 22259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:07.302031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.2:35651)
0928 06:39:07.302406 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:07.302481 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":83,"threads_started":1}
I20260928 06:39:07.498775 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:07.498303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.3:49309)
0928 06:39:07.498619 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:07.498683 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":73,"threads_started":1}
I20260928 06:39:08.147379 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:08.146913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.1:34371)
0928 06:39:08.147211 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:08.147276 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":92,"threads_started":1}
I20260928 06:39:08.303751 22262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:08.303275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.2:37141)
0928 06:39:08.303572 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:08.303629 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":99,"threads_started":1}
I20260928 06:39:08.500087 22263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:08.499637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.3:46179)
0928 06:39:08.499935 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:08.499996 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":80,"threads_started":1}
I20260928 06:39:09.001226 22264 raft_consensus.cc:482] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:39:09.001384 22264 raft_consensus.cc:504] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70347f0cca4f47d495db897103efcebb" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 46097 } } peers { permanent_uuid: "bd9107d3a54a4f998035e27180f4b8bc" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 36871 } } peers { permanent_uuid: "76f779950ca746809a2f124299de1c55" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 34999 } }
I20260928 06:39:09.001762 22264 leader_election.cc:288] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 70347f0cca4f47d495db897103efcebb (127.5.22.3:46097), bd9107d3a54a4f998035e27180f4b8bc (127.5.22.2:36871)
W20260928 06:39:09.002660 21843 leader_election.cc:341] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bd9107d3a54a4f998035e27180f4b8bc (127.5.22.2:36871): Not found: Tablet not found: 40b565bf318d495ea318e9094017789d
W20260928 06:39:09.002784 21846 leader_election.cc:341] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70347f0cca4f47d495db897103efcebb (127.5.22.3:46097): Not found: Tablet not found: 40b565bf318d495ea318e9094017789d
I20260928 06:39:09.002875 21846 leader_election.cc:302] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76f779950ca746809a2f124299de1c55; no voters: 70347f0cca4f47d495db897103efcebb, bd9107d3a54a4f998035e27180f4b8bc
I20260928 06:39:09.003038 22264 raft_consensus.cc:2726] T 40b565bf318d495ea318e9094017789d P 76f779950ca746809a2f124299de1c55 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:39:09.148662 22265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:09.148263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.1:36879)
0928 06:39:09.148514 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:09.148569 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":76,"threads_started":1}
I20260928 06:39:09.305085 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:09.304623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.2:57289)
0928 06:39:09.304927 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:09.304987 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":93,"threads_started":1}
I20260928 06:39:09.501442 22267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:09.501009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.3:60021)
0928 06:39:09.501283 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:09.501346 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":77,"threads_started":1}
I20260928 06:39:10.150265 22268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:10.149767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.1:47141)
0928 06:39:10.150082 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:10.150155 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":120,"threads_started":1}
I20260928 06:39:10.306520 22269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:10.305984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.2:46149)
0928 06:39:10.306245 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:10.306304 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
I20260928 06:39:10.502769 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:10.502261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.3:56023)
0928 06:39:10.502599 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:10.502653 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":163,"threads_started":1}
I20260928 06:39:11.151659 22271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:11.151199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.1:56989)
0928 06:39:11.151490 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:11.151562 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":100,"threads_started":1}
I20260928 06:39:11.307723 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:11.307298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.2:49161)
0928 06:39:11.307569 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:11.307625 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":81,"threads_started":1}
I20260928 06:39:11.504194 22273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:11.503544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.3:57105)
0928 06:39:11.504018 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:11.504083 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":85,"threads_started":1}
I20260928 06:39:12.152849 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:12.152425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.1:33965)
0928 06:39:12.152693 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:12.152752 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":78,"threads_started":1}
I20260928 06:39:12.309074 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:12.308653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.2:46057)
0928 06:39:12.308920 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:12.308980 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":90,"threads_started":1}
I20260928 06:39:12.505505 22276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:12.505035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.3:57163)
0928 06:39:12.505338 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:12.505402 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":96,"threads_started":1}
I20260928 06:39:13.154099 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:13.153668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.1:39555)
0928 06:39:13.153939 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:13.154000 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":86,"threads_started":1}
I20260928 06:39:13.310557 22278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:13.309967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.2:56795)
0928 06:39:13.310254 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:13.310449 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":86,"threads_started":1}
I20260928 06:39:13.506964 22279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:13.506445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.3:55905)
0928 06:39:13.506799 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:13.506866 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":106,"threads_started":1}
I20260928 06:39:14.155752 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:14.155300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.1:54747)
0928 06:39:14.155591 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:14.155656 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":84,"threads_started":1}
I20260928 06:39:14.312103 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:14.311572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.2:43329)
0928 06:39:14.311920 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:14.311988 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":94,"threads_started":1}
I20260928 06:39:14.500746 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 21786
I20260928 06:39:14.506969 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 21960
I20260928 06:39:14.509183 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:14.508761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:43541 (local address 127.5.22.3:58799)
0928 06:39:14.509031 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:14.509093 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.62:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":95,"threads_started":1}
I20260928 06:39:14.512677 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 22096
2026-09-28T06:39:14Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59854 ms)
[----------] 30 tests from SecurityITest (246342 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:14 dist-test-slave-j0qf krb5kdc[22285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:14 dist-test-slave-j0qf krb5kdc[22285](info): set up 2 sockets
Sep 28 06:39:14 dist-test-slave-j0qf krb5kdc[22285](info): commencing operation
krb5kdc: starting...
W20260928 06:39:15.896874 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:39:15 dist-test-slave-j0qf krb5kdc[22285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:39:16.044061 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:45223
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:45983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:45223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:16.157202 22298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:16.157473 22298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:16.157512 22298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:16.161494 22298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:16.161552 22298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:16.161572 22298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:16.161588 22298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:16.161603 22298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:16.166879 22298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:45983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:45223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:45223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:16.167842 22298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:16.168866 22298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:16.177526 22304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:16.177522 22303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:16.177522 22306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:16.177737 22298 server_base.cc:1034] running on GCE node
I20260928 06:39:16.178489 22298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:16.179706 22298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:16.180955 22298 hybrid_clock.cc:648] HybridClock initialized: now 1790577556180926 us; error 42 us; skew 500 ppm
Sep 28 06:39:16 dist-test-slave-j0qf krb5kdc[22285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577556, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:16.183717 22298 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:39:16.184675 22298 webserver.cc:460] Webserver started at http://127.5.22.62:35031/ using document root <none> and password file <none>
I20260928 06:39:16.185184 22298 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:16.185242 22298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:16.185418 22298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:16.187493 22298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "56f29147bbaa4e8ab10b72e62153694b"
format_stamp: "Formatted at 2026-09-28 06:39:16 on dist-test-slave-j0qf"
server_key: "d7200f37168371f1f1f0f0dedc1a4015"
server_key_iv: "38c25ebbbf7ffc4810eb5c0cbd1ceb7d"
server_key_version: "encryptionkey@0"
I20260928 06:39:16.188011 22298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "56f29147bbaa4e8ab10b72e62153694b"
format_stamp: "Formatted at 2026-09-28 06:39:16 on dist-test-slave-j0qf"
server_key: "d7200f37168371f1f1f0f0dedc1a4015"
server_key_iv: "38c25ebbbf7ffc4810eb5c0cbd1ceb7d"
server_key_version: "encryptionkey@0"
I20260928 06:39:16.191835 22298 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 06:39:16.194307 22313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:16.195747 22298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:39:16.195901 22298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "56f29147bbaa4e8ab10b72e62153694b"
format_stamp: "Formatted at 2026-09-28 06:39:16 on dist-test-slave-j0qf"
server_key: "d7200f37168371f1f1f0f0dedc1a4015"
server_key_iv: "38c25ebbbf7ffc4810eb5c0cbd1ceb7d"
server_key_version: "encryptionkey@0"
I20260928 06:39:16.196022 22298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:16.206105 22298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:16.209053 22298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:16.209259 22298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:16.216871 22298 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:45223
I20260928 06:39:16.216895 22366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:45223 every 8 connection(s)
I20260928 06:39:16.217806 22298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:16.220495 22367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:16.220705 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 22298
I20260928 06:39:16.220865 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:16.221174 5208 external_mini_cluster.cc:1428] Setting key fd0a251d3ca95bdbdbdadaf4f6306a3f
I20260928 06:39:16.227154 22367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56f29147bbaa4e8ab10b72e62153694b: Bootstrap starting.
I20260928 06:39:16.229650 22367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56f29147bbaa4e8ab10b72e62153694b: Neither blocks nor log segments found. Creating new log.
Sep 28 06:39:16 dist-test-slave-j0qf krb5kdc[22285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:39:16.230654 22367 log.cc:824] T 00000000000000000000000000000000 P 56f29147bbaa4e8ab10b72e62153694b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:16.232956 22367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56f29147bbaa4e8ab10b72e62153694b: No bootstrap required, opened a new log
I20260928 06:39:16.236397 22370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:16.222699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35904 (local address 127.5.22.62:45223)
0928 06:39:16.223141 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:16.223156 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:16.223192 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:39:16.223921 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:16.223938 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:16.224279 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:16.224640 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:16.225463 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:16.226533 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:16.227462 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:16.228031 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:16.228209 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:16.231048 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:16.231079 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:16.231099 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:16.231137 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:16.233297 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:16.233798 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:16.233802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:16.233807 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:16.233914 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:16.234299 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:16.234301 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:16.234302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:16.234780 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:16.234922 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:16.235345 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 06:39:16.235669 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":140,"threads_started":1}
I20260928 06:39:16.237151 22367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56f29147bbaa4e8ab10b72e62153694b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f29147bbaa4e8ab10b72e62153694b" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 45223 } }
I20260928 06:39:16.237406 22367 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56f29147bbaa4e8ab10b72e62153694b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:16.237465 22367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56f29147bbaa4e8ab10b72e62153694b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56f29147bbaa4e8ab10b72e62153694b, State: Initialized, Role: FOLLOWER
I20260928 06:39:16.237915 22367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56f29147bbaa4e8ab10b72e62153694b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f29147bbaa4e8ab10b72e62153694b" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 45223 } }
I20260928 06:39:16.238045 22367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56f29147bbaa4e8ab10b72e62153694b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:16.238138 22367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56f29147bbaa4e8ab10b72e62153694b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:16.238272 22367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56f29147bbaa4e8ab10b72e62153694b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:16.239208 22367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56f29147bbaa4e8ab10b72e62153694b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f29147bbaa4e8ab10b72e62153694b" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 45223 } }
I20260928 06:39:16.239559 22367 leader_election.cc:302] T 00000000000000000000000000000000 P 56f29147bbaa4e8ab10b72e62153694b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56f29147bbaa4e8ab10b72e62153694b; no voters:
I20260928 06:39:16.239877 22367 leader_election.cc:288] T 00000000000000000000000000000000 P 56f29147bbaa4e8ab10b72e62153694b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:16.240006 22372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56f29147bbaa4e8ab10b72e62153694b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:16.240255 22372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56f29147bbaa4e8ab10b72e62153694b [term 1 LEADER]: Becoming Leader. State: Replica: 56f29147bbaa4e8ab10b72e62153694b, State: Running, Role: LEADER
I20260928 06:39:16.240628 22372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56f29147bbaa4e8ab10b72e62153694b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f29147bbaa4e8ab10b72e62153694b" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 45223 } }
I20260928 06:39:16.241160 22367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56f29147bbaa4e8ab10b72e62153694b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:16.242358 22374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56f29147bbaa4e8ab10b72e62153694b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56f29147bbaa4e8ab10b72e62153694b. Latest consensus state: current_term: 1 leader_uuid: "56f29147bbaa4e8ab10b72e62153694b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f29147bbaa4e8ab10b72e62153694b" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 45223 } } }
I20260928 06:39:16.242501 22374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56f29147bbaa4e8ab10b72e62153694b [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:16.242837 22373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56f29147bbaa4e8ab10b72e62153694b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56f29147bbaa4e8ab10b72e62153694b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f29147bbaa4e8ab10b72e62153694b" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 45223 } } }
I20260928 06:39:16.242897 22381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:16.242991 22373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56f29147bbaa4e8ab10b72e62153694b [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:16.246354 22381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:16.252367 22381 catalog_manager.cc:1347] Generated new cluster ID: 745fbb5306e946b9923d95dced05c82e
I20260928 06:39:16.252446 22381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:16.258971 22381 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:16.259927 22381 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:16.266419 22381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56f29147bbaa4e8ab10b72e62153694b: Generated new TSK 0
I20260928 06:39:16.267117 22381 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.22.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
I20260928 06:39:16.307283 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:45223
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:45983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:16.420112 22393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:16.420403 22393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:16.420476 22393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:16.424476 22393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:16.424566 22393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:16.424671 22393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:39:16.429874 22393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:45983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:45223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:16.430977 22393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:16.432032 22393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:16.441900 22398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:16.442083 22393 server_base.cc:1034] running on GCE node
W20260928 06:39:16.441956 22399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:16.441900 22401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:16.442672 22393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:16.443349 22393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:16.444546 22393 hybrid_clock.cc:648] HybridClock initialized: now 1790577556444514 us; error 43 us; skew 500 ppm
Sep 28 06:39:16 dist-test-slave-j0qf krb5kdc[22285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577556, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:16.447271 22393 init.cc:377] Logged in from keytab as kudu/127.5.22.1@KRBTEST.COM (short username kudu)
I20260928 06:39:16.448146 22393 webserver.cc:460] Webserver started at http://127.5.22.1:32817/ using document root <none> and password file <none>
I20260928 06:39:16.448648 22393 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:16.448707 22393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:16.448873 22393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:16.450894 22393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "560cbb907173476e97128bff6889ac2a"
format_stamp: "Formatted at 2026-09-28 06:39:16 on dist-test-slave-j0qf"
server_key: "732a9574b8402dae1b8bbf3d18d25e48"
server_key_iv: "fb71a75cfcad0daa53e9c3a43f332508"
server_key_version: "encryptionkey@0"
I20260928 06:39:16.451387 22393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "560cbb907173476e97128bff6889ac2a"
format_stamp: "Formatted at 2026-09-28 06:39:16 on dist-test-slave-j0qf"
server_key: "732a9574b8402dae1b8bbf3d18d25e48"
server_key_iv: "fb71a75cfcad0daa53e9c3a43f332508"
server_key_version: "encryptionkey@0"
I20260928 06:39:16.455025 22393 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:39:16.457293 22408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:16.458534 22393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260928 06:39:16.458624 22393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "560cbb907173476e97128bff6889ac2a"
format_stamp: "Formatted at 2026-09-28 06:39:16 on dist-test-slave-j0qf"
server_key: "732a9574b8402dae1b8bbf3d18d25e48"
server_key_iv: "fb71a75cfcad0daa53e9c3a43f332508"
server_key_version: "encryptionkey@0"
I20260928 06:39:16.458755 22393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:16.473999 22393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:16.477560 22393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:16.477777 22393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:16.478477 22393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:16.479311 22393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:16.479362 22393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:16.479401 22393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:16.479446 22393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:16.488965 22393 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:35281
I20260928 06:39:16.489033 22522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:35281 every 8 connection(s)
I20260928 06:39:16.489929 22393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:39:16.494387 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 22393
I20260928 06:39:16.494553 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:39:16.494838 5208 external_mini_cluster.cc:1428] Setting key 5900bf5e926a078431a1951732f87462
Sep 28 06:39:16 dist-test-slave-j0qf krb5kdc[22285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577556, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:39:16.506003 22370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:16.492015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:33341 (local address 127.5.22.62:45223)
0928 06:39:16.492202 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:16.492207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:16.493302 (+ 1095us) server_negotiation.cc:407] Connection header received
0928 06:39:16.494282 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:16.494286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:16.494375 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:16.494473 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:16.496081 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:16.496877 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:16.498503 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:16.498723 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:16.498810 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:16.501751 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:16.501759 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:16.501766 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:16.501804 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:16.503718 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:16.504254 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:16.504259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:16.504261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:16.504368 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:16.504723 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:16.504729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:16.504731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:16.505010 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:16.505126 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:16.505662 (+ 536us) server_negotiation.cc:299] Negotiation successful
0928 06:39:16.505811 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
WARNING: no policy specified for kudu/127.5.22.2@KRBTEST.COM; defaulting to no policy
I20260928 06:39:16.506758 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:16.492338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:45223 (local address 127.5.22.1:33341)
0928 06:39:16.493047 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:16.493092 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:16.494023 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:16.494724 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:16.494732 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:16.495174 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:16.495918 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:16.495939 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:16.497031 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:16.497033 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:16.498362 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:16.498376 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:16.498905 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:16.498911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:16.499160 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:16.499912 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:16.499930 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:16.501589 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:16.503864 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:16.503869 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:16.503882 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:16.504135 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:16.504479 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:16.504481 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:16.504483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:16.504614 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:16.505121 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:16.505125 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:16.505388 (+ 263us) client_negotiation.cc:769] Sending connection context
0928 06:39:16.505546 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 06:39:16.505797 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":116,"threads_started":1}
I20260928 06:39:16.508211 22523 heartbeater.cc:347] Connected to a master server at 127.5.22.62:45223
I20260928 06:39:16.508440 22523 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:16.508894 22523 heartbeater.cc:511] Master 127.5.22.62:45223 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.2@KRBTEST.COM" created.
I20260928 06:39:16.510650 22331 ts_manager.cc:194] Registered new tserver with Master: 560cbb907173476e97128bff6889ac2a (127.5.22.1:35281)
I20260928 06:39:16.512007 22331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.1@KRBTEST.COM'} at 127.5.22.1:33341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
I20260928 06:39:16.525995 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:45223
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:45983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:16.636698 22528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:16.636955 22528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:16.636998 22528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:16.641281 22528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:16.641340 22528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:16.641398 22528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:39:16.646838 22528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:45983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:45223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:16.647877 22528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:16.648998 22528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:16.659092 22533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:16.659099 22536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:16.659090 22534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:16.659461 22528 server_base.cc:1034] running on GCE node
I20260928 06:39:16.659920 22528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:16.660570 22528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:16.661754 22528 hybrid_clock.cc:648] HybridClock initialized: now 1790577556661735 us; error 30 us; skew 500 ppm
Sep 28 06:39:16 dist-test-slave-j0qf krb5kdc[22285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577556, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:16.664798 22528 init.cc:377] Logged in from keytab as kudu/127.5.22.2@KRBTEST.COM (short username kudu)
I20260928 06:39:16.665709 22528 webserver.cc:460] Webserver started at http://127.5.22.2:41121/ using document root <none> and password file <none>
I20260928 06:39:16.666301 22528 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:16.666453 22528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:16.666689 22528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:16.668757 22528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "4439067a2a5c44cba5b54f6a43002726"
format_stamp: "Formatted at 2026-09-28 06:39:16 on dist-test-slave-j0qf"
server_key: "dba2063807948250875ec8e9cda8bed2"
server_key_iv: "37d7a1f42e5cd07eadb0a5ba7725ee1a"
server_key_version: "encryptionkey@0"
I20260928 06:39:16.669322 22528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "4439067a2a5c44cba5b54f6a43002726"
format_stamp: "Formatted at 2026-09-28 06:39:16 on dist-test-slave-j0qf"
server_key: "dba2063807948250875ec8e9cda8bed2"
server_key_iv: "37d7a1f42e5cd07eadb0a5ba7725ee1a"
server_key_version: "encryptionkey@0"
I20260928 06:39:16.673123 22528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260928 06:39:16.675781 22543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:16.677174 22528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 06:39:16.677384 22528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "4439067a2a5c44cba5b54f6a43002726"
format_stamp: "Formatted at 2026-09-28 06:39:16 on dist-test-slave-j0qf"
server_key: "dba2063807948250875ec8e9cda8bed2"
server_key_iv: "37d7a1f42e5cd07eadb0a5ba7725ee1a"
server_key_version: "encryptionkey@0"
I20260928 06:39:16.677513 22528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:16.700101 22528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:16.704067 22528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:16.704368 22528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:16.705238 22528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:16.706216 22528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:16.706264 22528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:16.706364 22528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:16.706405 22528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:16.715407 22528 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:40269
I20260928 06:39:16.715497 22657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:40269 every 8 connection(s)
I20260928 06:39:16.716346 22528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:39:16.722975 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 22528
I20260928 06:39:16.723135 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:39:16.723446 5208 external_mini_cluster.cc:1428] Setting key f1882c122dbea87aad74e2c3e78294f8
Sep 28 06:39:16 dist-test-slave-j0qf krb5kdc[22285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577556, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:39:16.730186 22370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:16.718205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:35895 (local address 127.5.22.62:45223)
0928 06:39:16.718404 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:16.718408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:16.719241 (+ 833us) server_negotiation.cc:407] Connection header received
0928 06:39:16.720147 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:16.720150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:16.720206 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:16.720318 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:16.721458 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:16.722013 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:16.723363 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:16.723639 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:16.723758 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:16.726522 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:16.726526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:16.726529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:16.726555 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:16.728077 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:16.728615 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:16.728618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:16.728625 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:16.728701 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:16.729070 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:16.729074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:16.729076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:16.729254 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:16.729366 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:16.729891 (+ 525us) server_negotiation.cc:299] Negotiation successful
0928 06:39:16.730014 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 06:39:16.731021 22660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:16.718643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:45223 (local address 127.5.22.2:35895)
0928 06:39:16.719093 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:16.719135 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:16.719937 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:16.720499 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:16.720506 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:16.720850 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:16.721299 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:16.721312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:16.722147 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:16.722151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:16.723233 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:16.723245 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:16.723796 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:16.723801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:16.723978 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:16.724618 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:16.724633 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:16.726288 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:16.728206 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:16.728212 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:16.728228 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:16.728496 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:16.728806 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:16.728809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:16.728811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:16.728954 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:16.729372 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:16.729380 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:16.729629 (+ 249us) client_negotiation.cc:769] Sending connection context
0928 06:39:16.729823 (+ 194us) client_negotiation.cc:240] Negotiation successful
0928 06:39:16.730146 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":118,"threads_started":1}
I20260928 06:39:16.732389 22658 heartbeater.cc:347] Connected to a master server at 127.5.22.62:45223
I20260928 06:39:16.732643 22658 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.22.3@KRBTEST.COM; defaulting to no policy
I20260928 06:39:16.733146 22658 heartbeater.cc:511] Master 127.5.22.62:45223 requested a full tablet report, sending...
I20260928 06:39:16.734189 22331 ts_manager.cc:194] Registered new tserver with Master: 4439067a2a5c44cba5b54f6a43002726 (127.5.22.2:40269)
I20260928 06:39:16.734782 22331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.2@KRBTEST.COM'} at 127.5.22.2:35895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
I20260928 06:39:16.755470 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:45223
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:45983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:16.870198 22663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:16.870558 22663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:16.870604 22663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:16.874868 22663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:16.874935 22663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:16.874998 22663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:39:16.880904 22663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:45983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:45223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:16.882185 22663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:16.883286 22663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:16.893766 22669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:16.893834 22671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:16.893971 22663 server_base.cc:1034] running on GCE node
W20260928 06:39:16.893790 22668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:16.894538 22663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:16.895246 22663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:16.896423 22663 hybrid_clock.cc:648] HybridClock initialized: now 1790577556896374 us; error 64 us; skew 500 ppm
Sep 28 06:39:16 dist-test-slave-j0qf krb5kdc[22285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577556, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:16.899485 22663 init.cc:377] Logged in from keytab as kudu/127.5.22.3@KRBTEST.COM (short username kudu)
I20260928 06:39:16.900424 22663 webserver.cc:460] Webserver started at http://127.5.22.3:40673/ using document root <none> and password file <none>
I20260928 06:39:16.900966 22663 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:16.901024 22663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:16.901190 22663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:16.903355 22663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "571227debed340d68934350fe0c0a667"
format_stamp: "Formatted at 2026-09-28 06:39:16 on dist-test-slave-j0qf"
server_key: "61b7f56bf76048d1d90de666610cf89e"
server_key_iv: "8ef6caf067d38b9780edc39ac0496b79"
server_key_version: "encryptionkey@0"
I20260928 06:39:16.903921 22663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "571227debed340d68934350fe0c0a667"
format_stamp: "Formatted at 2026-09-28 06:39:16 on dist-test-slave-j0qf"
server_key: "61b7f56bf76048d1d90de666610cf89e"
server_key_iv: "8ef6caf067d38b9780edc39ac0496b79"
server_key_version: "encryptionkey@0"
I20260928 06:39:16.907632 22663 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:39:16.910279 22678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:16.911641 22663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:39:16.911736 22663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "571227debed340d68934350fe0c0a667"
format_stamp: "Formatted at 2026-09-28 06:39:16 on dist-test-slave-j0qf"
server_key: "61b7f56bf76048d1d90de666610cf89e"
server_key_iv: "8ef6caf067d38b9780edc39ac0496b79"
server_key_version: "encryptionkey@0"
I20260928 06:39:16.911892 22663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:16.923590 22663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:16.927101 22663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:16.927315 22663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:16.927942 22663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:16.928776 22663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:16.928828 22663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:16.928901 22663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:16.928934 22663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:16.938407 22663 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:36005
I20260928 06:39:16.938436 22792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:36005 every 8 connection(s)
I20260928 06:39:16.939304 22663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:39:16.942454 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 22663
I20260928 06:39:16.942588 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
I20260928 06:39:16.942932 5208 external_mini_cluster.cc:1428] Setting key 4b9ddf41dd4a62fbf327cc4c4b26d2b4
Sep 28 06:39:16 dist-test-slave-j0qf krb5kdc[22285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577556, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:39:16.953425 22370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:16.941190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:52691 (local address 127.5.22.62:45223)
0928 06:39:16.941348 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:16.941353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:16.942905 (+ 1552us) server_negotiation.cc:407] Connection header received
0928 06:39:16.943926 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:16.943929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:16.943985 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:16.944069 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:16.945295 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:16.945833 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:16.947081 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:16.947309 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:16.947428 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:16.949912 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:16.949915 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:16.949918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:16.949948 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:16.951314 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:16.951914 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:16.951919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:16.951921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:16.952005 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:16.952374 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:16.952379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:16.952381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:16.952564 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:16.952681 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:16.953152 (+ 471us) server_negotiation.cc:299] Negotiation successful
0928 06:39:16.953267 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 06:39:16.954185 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:16.941640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:45223 (local address 127.5.22.3:52691)
0928 06:39:16.942735 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:16.942789 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:16.943704 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:16.944283 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:16.944290 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:16.944659 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:16.945137 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:16.945158 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:16.945970 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:16.945974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:16.946946 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:16.946956 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:16.947463 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:16.947467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:16.947644 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:16.948277 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:16.948293 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:16.949730 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:16.951501 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:16.951507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:16.951560 (+ 53us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:16.951801 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:16.952111 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:16.952114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:16.952116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:16.952268 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:16.952678 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:16.952684 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:16.952918 (+ 234us) client_negotiation.cc:769] Sending connection context
0928 06:39:16.953110 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 06:39:16.953427 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"thread_start_us":118,"threads_started":1}
I20260928 06:39:16.955286 22793 heartbeater.cc:347] Connected to a master server at 127.5.22.62:45223
I20260928 06:39:16.955539 22793 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:16.956136 22793 heartbeater.cc:511] Master 127.5.22.62:45223 requested a full tablet report, sending...
I20260928 06:39:16.957229 22331 ts_manager.cc:194] Registered new tserver with Master: 571227debed340d68934350fe0c0a667 (127.5.22.3:36005)
I20260928 06:39:16.957782 22331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.3@KRBTEST.COM'} at 127.5.22.3:52691
I20260928 06:39:16.969399 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:39:16.980715 22370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:16.972137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.5.22.62:45223)
0928 06:39:16.972535 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:16.972540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:16.972556 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:39:16.972615 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:16.972618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:16.972663 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:16.972743 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:16.973835 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:16.974522 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:16.975600 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:16.975816 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:16.975900 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:16.977508 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:16.977511 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:16.977513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:16.977536 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:16.979001 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:16.979418 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:16.979421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:16.979422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:16.979495 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:16.979746 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:16.979748 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:16.979749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:16.979968 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:16.980071 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:16.980434 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 06:39:16.980529 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261}
I20260928 06:39:16.983908 22331 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:39:16.987005 22331 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:39:17.002749 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:16.996514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42432 (local address 127.5.22.3:36005)
0928 06:39:16.997207 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:16.997222 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:16.997610 (+ 388us) server_negotiation.cc:407] Connection header received
0928 06:39:16.997855 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:16.997861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:16.998064 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:16.998229 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:16.998589 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:16.999681 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:17.001247 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:17.001910 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:17.002019 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:17.002165 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:17.002450 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 06:39:17.002552 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":156,"threads_started":1}
I20260928 06:39:17.002738 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:16.996374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:36005 (local address 127.0.0.1:42432)
0928 06:39:16.997468 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:16.997521 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:16.997663 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:16.998227 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:16.998233 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:16.998256 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:16.998461 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:16.998469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:16.999847 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:16.999854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:17.001095 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:17.001104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:17.002045 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:17.002049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:17.002166 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:17.002207 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 06:39:17.002416 (+ 209us) client_negotiation.cc:240] Negotiation successful
0928 06:39:17.002544 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"mutex_wait_us":115,"spinlock_wait_cycles":12544,"thread_start_us":103,"threads_started":1}
I20260928 06:39:17.003376 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:16.997156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:40269 (local address 127.0.0.1:47274)
0928 06:39:16.997908 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:16.997919 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:16.998033 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:16.998721 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:16.998724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:16.998741 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:16.998819 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:16.998824 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:17.000197 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:17.000201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:17.001621 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:17.001632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:17.002932 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:17.002935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:17.003033 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:17.003068 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 06:39:17.003145 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 06:39:17.003236 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":129,"thread_start_us":99,"threads_started":1}
I20260928 06:39:17.003575 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:16.997364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47274 (local address 127.5.22.2:40269)
0928 06:39:16.997837 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:16.997849 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:16.998010 (+ 161us) server_negotiation.cc:407] Connection header received
0928 06:39:16.998174 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:16.998182 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:16.998432 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:16.998592 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:16.998937 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:17.000036 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:17.001771 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:17.002607 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:17.002718 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:17.002864 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:17.003183 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 06:39:17.003279 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":141,"threads_started":1}
I20260928 06:39:17.003902 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:16.996373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:35281 (local address 127.0.0.1:44688)
0928 06:39:16.997470 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:16.997522 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:16.997663 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:16.998435 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:16.998437 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:16.998453 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:16.998574 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:16.998580 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:17.000197 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:17.000201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:17.001621 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:17.001632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:17.003213 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:17.003216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:17.003306 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:17.003329 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 06:39:17.003396 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:39:17.003753 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"mutex_wait_us":110,"spinlock_wait_cycles":2944,"thread_start_us":144,"threads_started":1}
I20260928 06:39:17.004617 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:16.996786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44688 (local address 127.5.22.1:35281)
0928 06:39:16.997759 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:16.997776 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:16.997801 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:39:16.997866 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:16.997870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:16.998083 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:16.998228 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:16.998714 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:17.000036 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:17.001774 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:17.002626 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:17.002738 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:17.004316 (+ 1578us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:17.004374 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 06:39:17.004469 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":104,"threads_started":1}
I20260928 06:39:17.006259 22591 tablet_service.cc:1467] Processing CreateTablet for tablet 736f3ccfd9bc4625807d173386fa194d (DEFAULT_TABLE table=test-table [id=7d340399897144a798b9df99ba871289]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:17.006361 22726 tablet_service.cc:1467] Processing CreateTablet for tablet 736f3ccfd9bc4625807d173386fa194d (DEFAULT_TABLE table=test-table [id=7d340399897144a798b9df99ba871289]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:17.007035 22456 tablet_service.cc:1467] Processing CreateTablet for tablet 736f3ccfd9bc4625807d173386fa194d (DEFAULT_TABLE table=test-table [id=7d340399897144a798b9df99ba871289]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:17.007447 22726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 736f3ccfd9bc4625807d173386fa194d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:17.007454 22591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 736f3ccfd9bc4625807d173386fa194d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:17.007936 22456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 736f3ccfd9bc4625807d173386fa194d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:17.013264 22810 tablet_bootstrap.cc:492] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726: Bootstrap starting.
I20260928 06:39:17.014853 22812 tablet_bootstrap.cc:492] T 736f3ccfd9bc4625807d173386fa194d P 560cbb907173476e97128bff6889ac2a: Bootstrap starting.
I20260928 06:39:17.016465 22810 tablet_bootstrap.cc:654] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:17.016831 22812 tablet_bootstrap.cc:654] T 736f3ccfd9bc4625807d173386fa194d P 560cbb907173476e97128bff6889ac2a: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:17.017285 22810 log.cc:824] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:17.017546 22812 log.cc:824] T 736f3ccfd9bc4625807d173386fa194d P 560cbb907173476e97128bff6889ac2a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:17.018208 22811 tablet_bootstrap.cc:492] T 736f3ccfd9bc4625807d173386fa194d P 571227debed340d68934350fe0c0a667: Bootstrap starting.
I20260928 06:39:17.019313 22812 tablet_bootstrap.cc:492] T 736f3ccfd9bc4625807d173386fa194d P 560cbb907173476e97128bff6889ac2a: No bootstrap required, opened a new log
I20260928 06:39:17.019441 22810 tablet_bootstrap.cc:492] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726: No bootstrap required, opened a new log
I20260928 06:39:17.019583 22812 ts_tablet_manager.cc:1397] T 736f3ccfd9bc4625807d173386fa194d P 560cbb907173476e97128bff6889ac2a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 06:39:17.019620 22810 ts_tablet_manager.cc:1397] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260928 06:39:17.020231 22811 tablet_bootstrap.cc:654] T 736f3ccfd9bc4625807d173386fa194d P 571227debed340d68934350fe0c0a667: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:17.021431 22811 log.cc:824] T 736f3ccfd9bc4625807d173386fa194d P 571227debed340d68934350fe0c0a667: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:17.023504 22811 tablet_bootstrap.cc:492] T 736f3ccfd9bc4625807d173386fa194d P 571227debed340d68934350fe0c0a667: No bootstrap required, opened a new log
I20260928 06:39:17.023703 22811 ts_tablet_manager.cc:1397] T 736f3ccfd9bc4625807d173386fa194d P 571227debed340d68934350fe0c0a667: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 06:39:17.023548 22810 raft_consensus.cc:348] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571227debed340d68934350fe0c0a667" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 36005 } } peers { permanent_uuid: "560cbb907173476e97128bff6889ac2a" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 35281 } } peers { permanent_uuid: "4439067a2a5c44cba5b54f6a43002726" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40269 } }
I20260928 06:39:17.023814 22810 raft_consensus.cc:374] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:17.023634 22812 raft_consensus.cc:348] T 736f3ccfd9bc4625807d173386fa194d P 560cbb907173476e97128bff6889ac2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571227debed340d68934350fe0c0a667" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 36005 } } peers { permanent_uuid: "560cbb907173476e97128bff6889ac2a" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 35281 } } peers { permanent_uuid: "4439067a2a5c44cba5b54f6a43002726" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40269 } }
I20260928 06:39:17.023871 22810 raft_consensus.cc:729] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4439067a2a5c44cba5b54f6a43002726, State: Initialized, Role: FOLLOWER
I20260928 06:39:17.023905 22812 raft_consensus.cc:374] T 736f3ccfd9bc4625807d173386fa194d P 560cbb907173476e97128bff6889ac2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:17.023968 22812 raft_consensus.cc:729] T 736f3ccfd9bc4625807d173386fa194d P 560cbb907173476e97128bff6889ac2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 560cbb907173476e97128bff6889ac2a, State: Initialized, Role: FOLLOWER
I20260928 06:39:17.024338 22810 consensus_queue.cc:260] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571227debed340d68934350fe0c0a667" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 36005 } } peers { permanent_uuid: "560cbb907173476e97128bff6889ac2a" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 35281 } } peers { permanent_uuid: "4439067a2a5c44cba5b54f6a43002726" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40269 } }
I20260928 06:39:17.024374 22812 consensus_queue.cc:260] T 736f3ccfd9bc4625807d173386fa194d P 560cbb907173476e97128bff6889ac2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571227debed340d68934350fe0c0a667" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 36005 } } peers { permanent_uuid: "560cbb907173476e97128bff6889ac2a" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 35281 } } peers { permanent_uuid: "4439067a2a5c44cba5b54f6a43002726" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40269 } }
I20260928 06:39:17.025070 22658 heartbeater.cc:503] Master 127.5.22.62:45223 was elected leader, sending a full tablet report...
I20260928 06:39:17.025452 22810 ts_tablet_manager.cc:1428] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 06:39:17.026425 22812 ts_tablet_manager.cc:1428] T 736f3ccfd9bc4625807d173386fa194d P 560cbb907173476e97128bff6889ac2a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 06:39:17.026804 22523 heartbeater.cc:503] Master 127.5.22.62:45223 was elected leader, sending a full tablet report...
I20260928 06:39:17.027472 22811 raft_consensus.cc:348] T 736f3ccfd9bc4625807d173386fa194d P 571227debed340d68934350fe0c0a667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571227debed340d68934350fe0c0a667" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 36005 } } peers { permanent_uuid: "560cbb907173476e97128bff6889ac2a" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 35281 } } peers { permanent_uuid: "4439067a2a5c44cba5b54f6a43002726" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40269 } }
I20260928 06:39:17.027766 22811 raft_consensus.cc:374] T 736f3ccfd9bc4625807d173386fa194d P 571227debed340d68934350fe0c0a667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:17.027828 22811 raft_consensus.cc:729] T 736f3ccfd9bc4625807d173386fa194d P 571227debed340d68934350fe0c0a667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571227debed340d68934350fe0c0a667, State: Initialized, Role: FOLLOWER
I20260928 06:39:17.028259 22811 consensus_queue.cc:260] T 736f3ccfd9bc4625807d173386fa194d P 571227debed340d68934350fe0c0a667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571227debed340d68934350fe0c0a667" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 36005 } } peers { permanent_uuid: "560cbb907173476e97128bff6889ac2a" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 35281 } } peers { permanent_uuid: "4439067a2a5c44cba5b54f6a43002726" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40269 } }
I20260928 06:39:17.028827 22793 heartbeater.cc:503] Master 127.5.22.62:45223 was elected leader, sending a full tablet report...
I20260928 06:39:17.029167 22811 ts_tablet_manager.cc:1428] T 736f3ccfd9bc4625807d173386fa194d P 571227debed340d68934350fe0c0a667: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
W20260928 06:39:17.190280 22794 tablet.cc:2369] T 736f3ccfd9bc4625807d173386fa194d P 571227debed340d68934350fe0c0a667: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:39:17.217576 22659 tablet.cc:2369] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:39:17.241416 22524 tablet.cc:2369] T 736f3ccfd9bc4625807d173386fa194d P 560cbb907173476e97128bff6889ac2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:39:17.481509 22817 raft_consensus.cc:482] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:39:17.481739 22817 raft_consensus.cc:504] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571227debed340d68934350fe0c0a667" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 36005 } } peers { permanent_uuid: "560cbb907173476e97128bff6889ac2a" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 35281 } } peers { permanent_uuid: "4439067a2a5c44cba5b54f6a43002726" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40269 } }
I20260928 06:39:17.483114 22817 leader_election.cc:288] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 571227debed340d68934350fe0c0a667 (127.5.22.3:36005), 560cbb907173476e97128bff6889ac2a (127.5.22.1:35281)
I20260928 06:39:17.488781 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:17.483576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:58233 (local address 127.5.22.1:35281)
0928 06:39:17.483713 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:17.483716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:17.483862 (+ 146us) server_negotiation.cc:407] Connection header received
0928 06:39:17.483964 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:17.483968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:17.484038 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:17.484126 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:17.484673 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:17.485597 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:17.487471 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:17.488094 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:17.488210 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:17.488347 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:17.488567 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 06:39:17.488646 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 06:39:17.488811 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:17.483485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:52537 (local address 127.5.22.3:36005)
0928 06:39:17.483619 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:17.483625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:17.483861 (+ 236us) server_negotiation.cc:407] Connection header received
0928 06:39:17.483964 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:17.483968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:17.484027 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:17.484127 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:17.484684 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:17.485542 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:17.487471 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:17.488094 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:17.488249 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:17.488309 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:17.488567 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 06:39:17.488646 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:39:17.488867 22821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:17.483388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:35281 (local address 127.5.22.2:58233)
0928 06:39:17.483707 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:17.483722 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:17.483872 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:17.484154 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:17.484157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:17.484194 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:17.484536 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:17.484542 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:17.485699 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:17.485702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:17.487288 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:17.487299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:17.488235 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:17.488238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:17.488371 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:17.488443 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 06:39:17.488528 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 06:39:17.488650 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":224,"spinlock_wait_cycles":60032,"thread_start_us":66,"threads_started":1}
I20260928 06:39:17.488895 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:17.483387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:36005 (local address 127.5.22.2:52537)
0928 06:39:17.483727 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:17.483740 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:17.483861 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:17.484154 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:17.484157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:17.484192 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:17.484530 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:17.484538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:17.485693 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:17.485697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:17.487288 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:17.487299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:17.488235 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:17.488238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:17.488436 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:17.488454 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 06:39:17.488528 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 06:39:17.488650 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":373,"spinlock_wait_cycles":26496,"thread_start_us":79,"threads_started":1}
I20260928 06:39:17.489701 22477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f3ccfd9bc4625807d173386fa194d" candidate_uuid: "4439067a2a5c44cba5b54f6a43002726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "560cbb907173476e97128bff6889ac2a" is_pre_election: true
I20260928 06:39:17.489995 22477 raft_consensus.cc:2454] T 736f3ccfd9bc4625807d173386fa194d P 560cbb907173476e97128bff6889ac2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4439067a2a5c44cba5b54f6a43002726 in term 0.
I20260928 06:39:17.490084 22746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f3ccfd9bc4625807d173386fa194d" candidate_uuid: "4439067a2a5c44cba5b54f6a43002726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571227debed340d68934350fe0c0a667" is_pre_election: true
I20260928 06:39:17.490370 22746 raft_consensus.cc:2454] T 736f3ccfd9bc4625807d173386fa194d P 571227debed340d68934350fe0c0a667 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4439067a2a5c44cba5b54f6a43002726 in term 0.
I20260928 06:39:17.490625 22546 leader_election.cc:302] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4439067a2a5c44cba5b54f6a43002726, 560cbb907173476e97128bff6889ac2a; no voters:
I20260928 06:39:17.490895 22817 raft_consensus.cc:2781] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:39:17.490978 22817 raft_consensus.cc:482] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:39:17.491026 22817 raft_consensus.cc:3037] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:17.492038 22817 raft_consensus.cc:504] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571227debed340d68934350fe0c0a667" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 36005 } } peers { permanent_uuid: "560cbb907173476e97128bff6889ac2a" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 35281 } } peers { permanent_uuid: "4439067a2a5c44cba5b54f6a43002726" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40269 } }
I20260928 06:39:17.492494 22817 leader_election.cc:288] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726 [CANDIDATE]: Term 1 election: Requested vote from peers 571227debed340d68934350fe0c0a667 (127.5.22.3:36005), 560cbb907173476e97128bff6889ac2a (127.5.22.1:35281)
I20260928 06:39:17.492862 22746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f3ccfd9bc4625807d173386fa194d" candidate_uuid: "4439067a2a5c44cba5b54f6a43002726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571227debed340d68934350fe0c0a667"
I20260928 06:39:17.492862 22477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f3ccfd9bc4625807d173386fa194d" candidate_uuid: "4439067a2a5c44cba5b54f6a43002726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "560cbb907173476e97128bff6889ac2a"
I20260928 06:39:17.492985 22477 raft_consensus.cc:3037] T 736f3ccfd9bc4625807d173386fa194d P 560cbb907173476e97128bff6889ac2a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:17.492986 22746 raft_consensus.cc:3037] T 736f3ccfd9bc4625807d173386fa194d P 571227debed340d68934350fe0c0a667 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:17.494231 22746 raft_consensus.cc:2454] T 736f3ccfd9bc4625807d173386fa194d P 571227debed340d68934350fe0c0a667 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4439067a2a5c44cba5b54f6a43002726 in term 1.
I20260928 06:39:17.494231 22477 raft_consensus.cc:2454] T 736f3ccfd9bc4625807d173386fa194d P 560cbb907173476e97128bff6889ac2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4439067a2a5c44cba5b54f6a43002726 in term 1.
I20260928 06:39:17.494551 22546 leader_election.cc:302] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4439067a2a5c44cba5b54f6a43002726, 560cbb907173476e97128bff6889ac2a; no voters:
I20260928 06:39:17.494758 22817 raft_consensus.cc:2781] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:17.495030 22817 raft_consensus.cc:686] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726 [term 1 LEADER]: Becoming Leader. State: Replica: 4439067a2a5c44cba5b54f6a43002726, State: Running, Role: LEADER
I20260928 06:39:17.495373 22817 consensus_queue.cc:237] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571227debed340d68934350fe0c0a667" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 36005 } } peers { permanent_uuid: "560cbb907173476e97128bff6889ac2a" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 35281 } } peers { permanent_uuid: "4439067a2a5c44cba5b54f6a43002726" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40269 } }
I20260928 06:39:17.498389 22331 catalog_manager.cc:5495] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4439067a2a5c44cba5b54f6a43002726 (127.5.22.2). New cstate: current_term: 1 leader_uuid: "4439067a2a5c44cba5b54f6a43002726" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571227debed340d68934350fe0c0a667" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 36005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "560cbb907173476e97128bff6889ac2a" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 35281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4439067a2a5c44cba5b54f6a43002726" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 40269 } health_report { overall_health: HEALTHY } } }
I20260928 06:39:17.568779 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:17.564830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47284 (local address 127.5.22.2:40269)
0928 06:39:17.565095 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:17.565098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:17.565113 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:39:17.565252 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:17.565254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:17.565371 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:17.565474 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:17.565764 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:17.566368 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:17.567307 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:17.567532 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:17.567621 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:17.567909 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:17.568334 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:17.568428 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:17.568496 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 06:39:17.568546 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":83,"threads_started":1}
I20260928 06:39:17.575083 22746 raft_consensus.cc:1261] T 736f3ccfd9bc4625807d173386fa194d P 571227debed340d68934350fe0c0a667 [term 1 FOLLOWER]: Refusing update from remote peer 4439067a2a5c44cba5b54f6a43002726: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:17.575089 22477 raft_consensus.cc:1261] T 736f3ccfd9bc4625807d173386fa194d P 560cbb907173476e97128bff6889ac2a [term 1 FOLLOWER]: Refusing update from remote peer 4439067a2a5c44cba5b54f6a43002726: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:17.575992 22822 consensus_queue.cc:1035] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571227debed340d68934350fe0c0a667" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 36005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:17.576198 22817 consensus_queue.cc:1035] T 736f3ccfd9bc4625807d173386fa194d P 4439067a2a5c44cba5b54f6a43002726 [LEADER]: Connected to new peer: Peer: permanent_uuid: "560cbb907173476e97128bff6889ac2a" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 35281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:17.585561 22830 mvcc.cc:205] Tried to move back new op lower bound from 7334205675819376640 to 7334205675503403008. Current Snapshot: MvccSnapshot[applied={T|T < 7334205675819376640}]
I20260928 06:39:17.593801 22330 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:39:17.593956 22330 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:39:17.596485 22330 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56f29147bbaa4e8ab10b72e62153694b: Sending DeleteTablet for 3 replicas of tablet 736f3ccfd9bc4625807d173386fa194d
I20260928 06:39:17.597325 22456 tablet_service.cc:1514] Processing DeleteTablet for tablet 736f3ccfd9bc4625807d173386fa194d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:17 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:44688
I20260928 06:39:17.597491 22726 tablet_service.cc:1514] Processing DeleteTablet for tablet 736f3ccfd9bc4625807d173386fa194d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:17 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:42432
I20260928 06:39:17.597815 22591 tablet_service.cc:1514] Processing DeleteTablet for tablet 736f3ccfd9bc4625807d173386fa194d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:17 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:47274
I20260928 06:39:17.597905 22842 tablet_replica.cc:330] T 736f3ccfd9bc4625807d173386fa194d P 560cbb907173476e97128bff6889ac2a: stopping tablet replica
I20260928 06:39:17.598067 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 22393
I20260928 06:39:17.602552 22843 tablet_replica.cc:330] T 736f3ccfd9bc4625807d173386fa194d P 571227debed340d68934350fe0c0a667: stopping tablet replica
I20260928 06:39:17.602896 22843 raft_consensus.cc:2229] T 736f3ccfd9bc4625807d173386fa194d P 571227debed340d68934350fe0c0a667 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:39:17.603222 22843 raft_consensus.cc:2258] T 736f3ccfd9bc4625807d173386fa194d P 571227debed340d68934350fe0c0a667 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:17.604633 22843 ts_tablet_manager.cc:1905] T 736f3ccfd9bc4625807d173386fa194d P 571227debed340d68934350fe0c0a667: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:39:17.605031 22317 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.22.1:35281 (error 108)
I20260928 06:39:17.605069 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 22528
I20260928 06:39:17.607234 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:17.606712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:35281 (local address 127.0.0.1:44698)
0928 06:39:17.607008 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:17.607089 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.1:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":57,"threads_started":1}
W20260928 06:39:17.607365 22317 catalog_manager.cc:4584] TS 560cbb907173476e97128bff6889ac2a (127.5.22.1:35281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 736f3ccfd9bc4625807d173386fa194d: Network error: Client connection negotiation failed: client connection to 127.5.22.1:35281: connect: Connection refused (error 111)
I20260928 06:39:17.608966 22843 ts_tablet_manager.cc:1918] T 736f3ccfd9bc4625807d173386fa194d P 571227debed340d68934350fe0c0a667: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:39:17.609084 22843 log.cc:1196] T 736f3ccfd9bc4625807d173386fa194d P 571227debed340d68934350fe0c0a667: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/736f3ccfd9bc4625807d173386fa194d
I20260928 06:39:17.609402 22843 ts_tablet_manager.cc:1939] T 736f3ccfd9bc4625807d173386fa194d P 571227debed340d68934350fe0c0a667: Deleting consensus metadata
I20260928 06:39:17.610908 22314 catalog_manager.cc:4841] TS 571227debed340d68934350fe0c0a667 (127.5.22.3:36005): tablet 736f3ccfd9bc4625807d173386fa194d (table test-table [id=7d340399897144a798b9df99ba871289]) successfully deleted
I20260928 06:39:17.612452 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 22663
I20260928 06:39:17.613824 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:17.613447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:40269 (local address 127.0.0.1:47294)
0928 06:39:17.613646 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:17.613708 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.2:40269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 06:39:17.618661 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 22298
2026-09-28T06:39:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2789 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T06:39:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:17Z Disabled control of system clock
I20260928 06:39:17.650595 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:41051
--webserver_interface=127.5.22.62
--webserver_port=0
--builtin_ntp_servers=127.5.22.20:45589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:41051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:39:17.765609 22850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:17.765868 22850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:17.765904 22850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:17.769945 22850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:17.770015 22850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:17.770030 22850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:17.770047 22850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:17.770063 22850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:17.774935 22850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:45589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:41051
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:41051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:17.776136 22850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:17.777559 22850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:17.787007 22855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:17.787012 22858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:17.787029 22856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:17.787376 22850 server_base.cc:1034] running on GCE node
I20260928 06:39:17.787922 22850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:17.789130 22850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:17.790397 22850 hybrid_clock.cc:648] HybridClock initialized: now 1790577557790360 us; error 46 us; skew 500 ppm
I20260928 06:39:17.792418 22850 webserver.cc:460] Webserver started at http://127.5.22.62:46639/ using document root <none> and password file <none>
I20260928 06:39:17.793013 22850 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:17.793112 22850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:17.793329 22850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:17.795481 22850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "08cb7721e35a447b8be4397e1124027c"
format_stamp: "Formatted at 2026-09-28 06:39:17 on dist-test-slave-j0qf"
server_key: "acf3040d75ec564e6c532b384971d525"
server_key_iv: "897135989c47b762895c5fe57c206d28"
server_key_version: "encryptionkey@0"
I20260928 06:39:17.796114 22850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "08cb7721e35a447b8be4397e1124027c"
format_stamp: "Formatted at 2026-09-28 06:39:17 on dist-test-slave-j0qf"
server_key: "acf3040d75ec564e6c532b384971d525"
server_key_iv: "897135989c47b762895c5fe57c206d28"
server_key_version: "encryptionkey@0"
I20260928 06:39:17.799876 22850 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:39:17.802297 22864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:17.803720 22850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:39:17.803874 22850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "08cb7721e35a447b8be4397e1124027c"
format_stamp: "Formatted at 2026-09-28 06:39:17 on dist-test-slave-j0qf"
server_key: "acf3040d75ec564e6c532b384971d525"
server_key_iv: "897135989c47b762895c5fe57c206d28"
server_key_version: "encryptionkey@0"
I20260928 06:39:17.803999 22850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:17.815894 22850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:17.816638 22850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:17.816854 22850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:17.824064 22850 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:41051
I20260928 06:39:17.824074 22917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:41051 every 8 connection(s)
I20260928 06:39:17.825008 22850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:17.826594 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 22850
I20260928 06:39:17.826750 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:17.827035 5208 external_mini_cluster.cc:1428] Setting key 86d92e275fc67c6446790112635bff0f
I20260928 06:39:17.828536 22918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:17.834810 22918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08cb7721e35a447b8be4397e1124027c: Bootstrap starting.
I20260928 06:39:17.836576 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:17.828442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49548 (local address 127.5.22.62:41051)
0928 06:39:17.829092 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:17.829107 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:17.829142 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:39:17.829792 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:17.829810 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:17.830184 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:17.830538 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:17.830875 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:17.831839 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:17.833094 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:17.833582 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:17.833700 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:17.834720 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:17.834725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:17.834738 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:17.835219 (+ 481us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:17.835286 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:17.835317 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:17.835399 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:17.835576 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 06:39:17.835894 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":186,"threads_started":1}
I20260928 06:39:17.837146 22918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08cb7721e35a447b8be4397e1124027c: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:17.838064 22918 log.cc:824] T 00000000000000000000000000000000 P 08cb7721e35a447b8be4397e1124027c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:17.840097 22918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08cb7721e35a447b8be4397e1124027c: No bootstrap required, opened a new log
I20260928 06:39:17.843972 22918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08cb7721e35a447b8be4397e1124027c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cb7721e35a447b8be4397e1124027c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 41051 } }
I20260928 06:39:17.844182 22918 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08cb7721e35a447b8be4397e1124027c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:17.844218 22918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08cb7721e35a447b8be4397e1124027c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08cb7721e35a447b8be4397e1124027c, State: Initialized, Role: FOLLOWER
I20260928 06:39:17.844614 22918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08cb7721e35a447b8be4397e1124027c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cb7721e35a447b8be4397e1124027c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 41051 } }
I20260928 06:39:17.844738 22918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08cb7721e35a447b8be4397e1124027c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:17.844784 22918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08cb7721e35a447b8be4397e1124027c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:17.844854 22918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08cb7721e35a447b8be4397e1124027c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:17.845659 22918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08cb7721e35a447b8be4397e1124027c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cb7721e35a447b8be4397e1124027c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 41051 } }
I20260928 06:39:17.845939 22918 leader_election.cc:302] T 00000000000000000000000000000000 P 08cb7721e35a447b8be4397e1124027c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08cb7721e35a447b8be4397e1124027c; no voters:
I20260928 06:39:17.846184 22918 leader_election.cc:288] T 00000000000000000000000000000000 P 08cb7721e35a447b8be4397e1124027c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:17.846323 22923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08cb7721e35a447b8be4397e1124027c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:17.846608 22923 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08cb7721e35a447b8be4397e1124027c [term 1 LEADER]: Becoming Leader. State: Replica: 08cb7721e35a447b8be4397e1124027c, State: Running, Role: LEADER
I20260928 06:39:17.846946 22923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08cb7721e35a447b8be4397e1124027c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cb7721e35a447b8be4397e1124027c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 41051 } }
I20260928 06:39:17.847460 22918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08cb7721e35a447b8be4397e1124027c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:17.848716 22925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08cb7721e35a447b8be4397e1124027c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08cb7721e35a447b8be4397e1124027c. Latest consensus state: current_term: 1 leader_uuid: "08cb7721e35a447b8be4397e1124027c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cb7721e35a447b8be4397e1124027c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 41051 } } }
I20260928 06:39:17.848671 22924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08cb7721e35a447b8be4397e1124027c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08cb7721e35a447b8be4397e1124027c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cb7721e35a447b8be4397e1124027c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 41051 } } }
I20260928 06:39:17.848809 22925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08cb7721e35a447b8be4397e1124027c [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:17.848809 22924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08cb7721e35a447b8be4397e1124027c [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:17.849242 22932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:17.852304 22932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:17.858301 22932 catalog_manager.cc:1347] Generated new cluster ID: 5796831f4cca40d2b2fe869f8fea69f4
I20260928 06:39:17.858392 22932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:17.877555 22932 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:17.878885 22932 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:17.887537 22932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08cb7721e35a447b8be4397e1124027c: Generated new TSK 0
I20260928 06:39:17.888363 22932 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:17.895673 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:41051
--builtin_ntp_servers=127.5.22.20:45589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:39:18.010608 22942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:18.010881 22942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:18.010921 22942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:18.015925 22942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:18.016009 22942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:18.016070 22942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:39:18.020990 22942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:45589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:41051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:18.022106 22942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:18.023537 22942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:18.033309 22950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:18.033310 22948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:18.033412 22947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:18.033697 22942 server_base.cc:1034] running on GCE node
I20260928 06:39:18.034117 22942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:18.034883 22942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:18.036090 22942 hybrid_clock.cc:648] HybridClock initialized: now 1790577558036073 us; error 35 us; skew 500 ppm
I20260928 06:39:18.038066 22942 webserver.cc:460] Webserver started at http://127.5.22.1:41815/ using document root <none> and password file <none>
I20260928 06:39:18.038735 22942 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:18.038841 22942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:18.039101 22942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:18.041137 22942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "0b00509ca863446eb188b2595e524b4d"
format_stamp: "Formatted at 2026-09-28 06:39:18 on dist-test-slave-j0qf"
server_key: "359852c707e751c328896a7eba24934c"
server_key_iv: "7b1c150ed11f987b6a0317dd1f785fd6"
server_key_version: "encryptionkey@0"
I20260928 06:39:18.041766 22942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "0b00509ca863446eb188b2595e524b4d"
format_stamp: "Formatted at 2026-09-28 06:39:18 on dist-test-slave-j0qf"
server_key: "359852c707e751c328896a7eba24934c"
server_key_iv: "7b1c150ed11f987b6a0317dd1f785fd6"
server_key_version: "encryptionkey@0"
I20260928 06:39:18.045439 22942 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 06:39:18.047993 22956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:18.049233 22942 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:39:18.049371 22942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "0b00509ca863446eb188b2595e524b4d"
format_stamp: "Formatted at 2026-09-28 06:39:18 on dist-test-slave-j0qf"
server_key: "359852c707e751c328896a7eba24934c"
server_key_iv: "7b1c150ed11f987b6a0317dd1f785fd6"
server_key_version: "encryptionkey@0"
I20260928 06:39:18.049482 22942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:18.061457 22942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:18.062207 22942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:18.062465 22942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:18.063041 22942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:18.063964 22942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:18.064041 22942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:18.064107 22942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:18.064169 22942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:18.073521 22942 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:43677
I20260928 06:39:18.073573 23070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:43677 every 8 connection(s)
I20260928 06:39:18.074515 22942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:39:18.082506 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 22942
I20260928 06:39:18.082644 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:39:18.083002 5208 external_mini_cluster.cc:1428] Setting key 1fb278ed2dcd7be902a34054900eb966
I20260928 06:39:18.084299 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:18.076423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:42253 (local address 127.5.22.62:41051)
0928 06:39:18.076562 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:18.076566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:18.077617 (+ 1051us) server_negotiation.cc:407] Connection header received
0928 06:39:18.079172 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:18.079175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:18.079251 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:18.079363 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:18.080269 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.080814 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.081967 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.082196 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.082325 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:18.083778 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:18.083782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:18.083785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:18.083797 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:18.083821 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:18.083832 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:18.083916 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:18.084099 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 06:39:18.084173 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 06:39:18.085232 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:18.076788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:41051 (local address 127.5.22.1:42253)
0928 06:39:18.077460 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:18.077502 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:18.078951 (+ 1449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:18.079540 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:18.079546 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:18.079630 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:18.080116 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:18.080128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.080955 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.080964 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:18.081836 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:18.081844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.082388 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.082393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:18.082558 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:18.083465 (+ 907us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:39:18.083469 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:18.083524 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:39:18.083608 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:18.083940 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:18.083949 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:18.083956 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 06:39:18.084088 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 06:39:18.084382 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":116,"threads_started":1}
I20260928 06:39:18.085314 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:41051
--builtin_ntp_servers=127.5.22.20:45589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:39:18.086701 23071 heartbeater.cc:347] Connected to a master server at 127.5.22.62:41051
I20260928 06:39:18.086980 23071 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:18.087611 23071 heartbeater.cc:511] Master 127.5.22.62:41051 requested a full tablet report, sending...
I20260928 06:39:18.089313 22881 ts_manager.cc:194] Registered new tserver with Master: 0b00509ca863446eb188b2595e524b4d (127.5.22.1:43677)
I20260928 06:39:18.090570 22881 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.22.1:42253
W20260928 06:39:18.197650 23074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:18.197955 23074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:18.198031 23074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:18.202080 23074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:18.202168 23074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:18.202252 23074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:39:18.207039 23074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:45589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:41051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:18.208155 23074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:18.209581 23074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:18.219487 23079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:18.219487 23080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:18.219651 23074 server_base.cc:1034] running on GCE node
W20260928 06:39:18.219488 23082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:18.220263 23074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:18.220850 23074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:18.222015 23074 hybrid_clock.cc:648] HybridClock initialized: now 1790577558221937 us; error 92 us; skew 500 ppm
I20260928 06:39:18.223932 23074 webserver.cc:460] Webserver started at http://127.5.22.2:46143/ using document root <none> and password file <none>
I20260928 06:39:18.224540 23074 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:18.224606 23074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:18.224824 23074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:18.226928 23074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "8322e4d55c1e41cdaa8a33557c47379a"
format_stamp: "Formatted at 2026-09-28 06:39:18 on dist-test-slave-j0qf"
server_key: "e88ee667972ef67160fed7f29c333b05"
server_key_iv: "fac4c25b0000ae285ad46a45582e4970"
server_key_version: "encryptionkey@0"
I20260928 06:39:18.227532 23074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "8322e4d55c1e41cdaa8a33557c47379a"
format_stamp: "Formatted at 2026-09-28 06:39:18 on dist-test-slave-j0qf"
server_key: "e88ee667972ef67160fed7f29c333b05"
server_key_iv: "fac4c25b0000ae285ad46a45582e4970"
server_key_version: "encryptionkey@0"
I20260928 06:39:18.231281 23074 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 06:39:18.233762 23088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:18.235019 23074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:39:18.235148 23074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "8322e4d55c1e41cdaa8a33557c47379a"
format_stamp: "Formatted at 2026-09-28 06:39:18 on dist-test-slave-j0qf"
server_key: "e88ee667972ef67160fed7f29c333b05"
server_key_iv: "fac4c25b0000ae285ad46a45582e4970"
server_key_version: "encryptionkey@0"
I20260928 06:39:18.235267 23074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:18.250133 23074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:18.250842 23074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:18.251045 23074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:18.251636 23074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:18.252673 23074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:18.252731 23074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:18.252772 23074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:18.252851 23074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:18.261926 23074 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:43941
I20260928 06:39:18.261958 23202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:43941 every 8 connection(s)
I20260928 06:39:18.262887 23074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:39:18.271697 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:18.264665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:58515 (local address 127.5.22.62:41051)
0928 06:39:18.264848 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:18.264853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:18.265897 (+ 1044us) server_negotiation.cc:407] Connection header received
0928 06:39:18.267288 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:18.267290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:18.267375 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:18.267450 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:18.268338 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.268853 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.269959 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.270166 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.270241 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:18.271259 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:18.271262 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:18.271264 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:18.271276 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:18.271295 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:18.271304 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:18.271389 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:18.271534 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 06:39:18.271586 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 06:39:18.272513 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:18.265083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:41051 (local address 127.5.22.2:58515)
0928 06:39:18.265733 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:18.265775 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:18.267096 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:18.267614 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:18.267620 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:18.267700 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:18.268179 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:18.268190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.268979 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.268982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:18.269848 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:18.269856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.270299 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.270302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:18.270452 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:18.271018 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:39:18.271020 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:18.271057 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:39:18.271114 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:18.271407 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:18.271414 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:18.271418 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:39:18.271507 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:39:18.271711 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":116,"threads_started":1}
I20260928 06:39:18.272895 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 23074
I20260928 06:39:18.273028 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:39:18.273331 5208 external_mini_cluster.cc:1428] Setting key c2a4cc4dbd04dc5b4ad4fdd8b619112f
I20260928 06:39:18.273864 23203 heartbeater.cc:347] Connected to a master server at 127.5.22.62:41051
I20260928 06:39:18.274114 23203 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:18.274746 23203 heartbeater.cc:511] Master 127.5.22.62:41051 requested a full tablet report, sending...
I20260928 06:39:18.275611 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:41051
--builtin_ntp_servers=127.5.22.20:45589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:39:18.275841 22881 ts_manager.cc:194] Registered new tserver with Master: 8322e4d55c1e41cdaa8a33557c47379a (127.5.22.2:43941)
I20260928 06:39:18.276471 22881 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.22.2:58515
W20260928 06:39:18.385245 23206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:18.385488 23206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:18.385524 23206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:18.389866 23206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:18.389922 23206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:18.390015 23206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:39:18.394932 23206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:45589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:41051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:18.395973 23206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:18.397310 23206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:18.407045 23214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:18.407056 23211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:18.407137 23212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:18.407188 23206 server_base.cc:1034] running on GCE node
I20260928 06:39:18.407689 23206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:18.408298 23206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:18.409492 23206 hybrid_clock.cc:648] HybridClock initialized: now 1790577558409476 us; error 41 us; skew 500 ppm
I20260928 06:39:18.411396 23206 webserver.cc:460] Webserver started at http://127.5.22.3:45353/ using document root <none> and password file <none>
I20260928 06:39:18.411963 23206 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:18.412060 23206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:18.412322 23206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:18.414296 23206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "d7068e357a1d49df89eef4fef0908d8c"
format_stamp: "Formatted at 2026-09-28 06:39:18 on dist-test-slave-j0qf"
server_key: "bbbd1295e36ba76c09c625c83f18707d"
server_key_iv: "59cfd8e4e1c2224e881cc7e38e4dd7b7"
server_key_version: "encryptionkey@0"
I20260928 06:39:18.414925 23206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "d7068e357a1d49df89eef4fef0908d8c"
format_stamp: "Formatted at 2026-09-28 06:39:18 on dist-test-slave-j0qf"
server_key: "bbbd1295e36ba76c09c625c83f18707d"
server_key_iv: "59cfd8e4e1c2224e881cc7e38e4dd7b7"
server_key_version: "encryptionkey@0"
I20260928 06:39:18.418776 23206 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 06:39:18.421078 23220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:18.422307 23206 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:39:18.422490 23206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "d7068e357a1d49df89eef4fef0908d8c"
format_stamp: "Formatted at 2026-09-28 06:39:18 on dist-test-slave-j0qf"
server_key: "bbbd1295e36ba76c09c625c83f18707d"
server_key_iv: "59cfd8e4e1c2224e881cc7e38e4dd7b7"
server_key_version: "encryptionkey@0"
I20260928 06:39:18.422598 23206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:18.442411 23206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:18.443145 23206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:18.443341 23206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:18.443938 23206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:18.444869 23206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:18.444916 23206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:18.444981 23206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:18.445013 23206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:18.454689 23206 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:36879
I20260928 06:39:18.454718 23334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:36879 every 8 connection(s)
I20260928 06:39:18.455611 23206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:39:18.462515 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 23206
I20260928 06:39:18.462610 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
I20260928 06:39:18.462885 5208 external_mini_cluster.cc:1428] Setting key 919738bfc9418d4623ec0fe215325a57
I20260928 06:39:18.466562 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:18.457596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:54649 (local address 127.5.22.62:41051)
0928 06:39:18.457778 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:18.457781 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:18.458897 (+ 1116us) server_negotiation.cc:407] Connection header received
0928 06:39:18.460549 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:18.460553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:18.460621 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:18.460739 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:18.461762 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.462381 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.464069 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.464406 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.464546 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:18.465891 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:18.465895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:18.465899 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:18.465918 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:18.465939 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:18.465948 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:18.466036 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:18.466345 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 06:39:18.466408 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 06:39:18.467443 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:18.457971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:41051 (local address 127.5.22.3:54649)
0928 06:39:18.458711 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:18.458758 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:18.460283 (+ 1525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:18.460953 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:18.460961 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:18.461052 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:18.461582 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:18.461595 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.462544 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.462552 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:18.463931 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:18.463941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.464591 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.464595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:18.464791 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:18.465581 (+ 790us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:39:18.465583 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:18.465629 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:39:18.465714 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:18.466061 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:18.466075 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:18.466086 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 06:39:18.466210 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 06:39:18.466584 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":118,"threads_started":1}
I20260928 06:39:18.468745 23335 heartbeater.cc:347] Connected to a master server at 127.5.22.62:41051
I20260928 06:39:18.468976 23335 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:18.469493 23335 heartbeater.cc:511] Master 127.5.22.62:41051 requested a full tablet report, sending...
I20260928 06:39:18.470656 22881 ts_manager.cc:194] Registered new tserver with Master: d7068e357a1d49df89eef4fef0908d8c (127.5.22.3:36879)
I20260928 06:39:18.471300 22881 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.22.3:54649
I20260928 06:39:18.477295 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:39:18.484201 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:18.479514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59500 (local address 127.5.22.62:41051)
0928 06:39:18.479732 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:18.479738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:18.479921 (+ 183us) server_negotiation.cc:407] Connection header received
0928 06:39:18.480099 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:18.480102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:18.480168 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:18.480273 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:18.480600 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.481255 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.482257 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.482584 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.482721 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:18.483680 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:18.483685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:18.483687 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:18.483701 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:18.483725 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:18.483741 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:18.483833 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:18.483973 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 06:39:18.484032 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260928 06:39:18.487449 22881 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:39:18.490692 22881 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:39:18.506487 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:18.500365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:43677 (local address 127.0.0.1:55390)
0928 06:39:18.501003 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:18.501019 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:18.501118 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:18.501823 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:18.501825 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:18.501861 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:18.502033 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:18.502040 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.503228 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.503234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:18.504678 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:18.504687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.505903 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.505907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:18.506013 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:18.506062 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 06:39:18.506166 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 06:39:18.506286 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":69,"spinlock_wait_cycles":9344,"thread_start_us":119,"threads_started":1}
I20260928 06:39:18.506688 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:18.500034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:43941 (local address 127.0.0.1:44324)
0928 06:39:18.500678 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:18.500730 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:18.500880 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:18.501815 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:18.501822 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:18.501865 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:18.502094 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:18.502104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.503703 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.503707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:18.505085 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:18.505097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.506152 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.506159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:18.506280 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:18.506304 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 06:39:18.506416 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 06:39:18.506495 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":85,"spinlock_wait_cycles":2560,"thread_start_us":94,"threads_started":1}
I20260928 06:39:18.506963 23347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:18.500173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44324 (local address 127.5.22.2:43941)
0928 06:39:18.501350 (+ 1177us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:18.501362 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:18.501386 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:39:18.501457 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:18.501462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:18.501619 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:18.502079 (+ 460us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:18.502239 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.503542 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.505238 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.506013 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.506238 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:18.506447 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:18.506500 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 06:39:18.506786 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1061,"thread_start_us":115,"threads_started":1}
I20260928 06:39:18.507011 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:18.500547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55390 (local address 127.5.22.1:43677)
0928 06:39:18.501066 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:18.501081 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:18.501109 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:39:18.501318 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:18.501325 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:18.501553 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:18.501725 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:18.502181 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.503105 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.504836 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.505531 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.505633 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:18.505835 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:18.506464 (+ 629us) server_negotiation.cc:299] Negotiation successful
0928 06:39:18.506578 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":158,"threads_started":1}
I20260928 06:39:18.507630 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:18.501233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42028 (local address 127.5.22.3:36879)
0928 06:39:18.501646 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:18.501662 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:18.501718 (+ 56us) server_negotiation.cc:407] Connection header received
0928 06:39:18.501871 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:18.501876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:18.502049 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:18.502192 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:18.502594 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.503794 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.505387 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.505977 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.506124 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:18.506321 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:18.507380 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0928 06:39:18.507479 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":94,"threads_started":1}
I20260928 06:39:18.507893 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:18.500780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:36879 (local address 127.0.0.1:42028)
0928 06:39:18.501622 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:18.501635 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:18.501717 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:18.502366 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:18.502369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:18.502386 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:18.502457 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:18.502463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.503925 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.503929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:18.505248 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:18.505257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.506126 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.506130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:18.506967 (+ 837us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:18.506991 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 06:39:18.507072 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 06:39:18.507750 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":6,"thread_start_us":93,"threads_started":1}
I20260928 06:39:18.509969 23137 tablet_service.cc:1467] Processing CreateTablet for tablet 038e44ad16b74609835844ee46bc48bf (DEFAULT_TABLE table=test-table [id=4db20eb54c954ca28823bcfae6744357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:18.509956 23005 tablet_service.cc:1467] Processing CreateTablet for tablet 038e44ad16b74609835844ee46bc48bf (DEFAULT_TABLE table=test-table [id=4db20eb54c954ca28823bcfae6744357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:18.510519 23269 tablet_service.cc:1467] Processing CreateTablet for tablet 038e44ad16b74609835844ee46bc48bf (DEFAULT_TABLE table=test-table [id=4db20eb54c954ca28823bcfae6744357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:18.511066 23137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038e44ad16b74609835844ee46bc48bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:18.511086 23005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038e44ad16b74609835844ee46bc48bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:18.512712 23269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038e44ad16b74609835844ee46bc48bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:18.518232 23352 tablet_bootstrap.cc:492] T 038e44ad16b74609835844ee46bc48bf P 0b00509ca863446eb188b2595e524b4d: Bootstrap starting.
I20260928 06:39:18.519619 23353 tablet_bootstrap.cc:492] T 038e44ad16b74609835844ee46bc48bf P 8322e4d55c1e41cdaa8a33557c47379a: Bootstrap starting.
I20260928 06:39:18.519848 23354 tablet_bootstrap.cc:492] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c: Bootstrap starting.
I20260928 06:39:18.521377 23353 tablet_bootstrap.cc:654] T 038e44ad16b74609835844ee46bc48bf P 8322e4d55c1e41cdaa8a33557c47379a: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:18.521435 23352 tablet_bootstrap.cc:654] T 038e44ad16b74609835844ee46bc48bf P 0b00509ca863446eb188b2595e524b4d: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:18.521986 23354 tablet_bootstrap.cc:654] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:18.522297 23353 log.cc:824] T 038e44ad16b74609835844ee46bc48bf P 8322e4d55c1e41cdaa8a33557c47379a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:18.522931 23354 log.cc:824] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:18.522931 23352 log.cc:824] T 038e44ad16b74609835844ee46bc48bf P 0b00509ca863446eb188b2595e524b4d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:18.524586 23353 tablet_bootstrap.cc:492] T 038e44ad16b74609835844ee46bc48bf P 8322e4d55c1e41cdaa8a33557c47379a: No bootstrap required, opened a new log
I20260928 06:39:18.524786 23354 tablet_bootstrap.cc:492] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c: No bootstrap required, opened a new log
I20260928 06:39:18.524787 23353 ts_tablet_manager.cc:1397] T 038e44ad16b74609835844ee46bc48bf P 8322e4d55c1e41cdaa8a33557c47379a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 06:39:18.525008 23354 ts_tablet_manager.cc:1397] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 06:39:18.525058 23352 tablet_bootstrap.cc:492] T 038e44ad16b74609835844ee46bc48bf P 0b00509ca863446eb188b2595e524b4d: No bootstrap required, opened a new log
I20260928 06:39:18.525238 23352 ts_tablet_manager.cc:1397] T 038e44ad16b74609835844ee46bc48bf P 0b00509ca863446eb188b2595e524b4d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260928 06:39:18.528811 23354 raft_consensus.cc:348] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8322e4d55c1e41cdaa8a33557c47379a" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 43941 } } peers { permanent_uuid: "d7068e357a1d49df89eef4fef0908d8c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 36879 } } peers { permanent_uuid: "0b00509ca863446eb188b2595e524b4d" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43677 } }
I20260928 06:39:18.529107 23354 raft_consensus.cc:374] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:18.529173 23354 raft_consensus.cc:729] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7068e357a1d49df89eef4fef0908d8c, State: Initialized, Role: FOLLOWER
I20260928 06:39:18.529296 23353 raft_consensus.cc:348] T 038e44ad16b74609835844ee46bc48bf P 8322e4d55c1e41cdaa8a33557c47379a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8322e4d55c1e41cdaa8a33557c47379a" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 43941 } } peers { permanent_uuid: "d7068e357a1d49df89eef4fef0908d8c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 36879 } } peers { permanent_uuid: "0b00509ca863446eb188b2595e524b4d" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43677 } }
I20260928 06:39:18.529547 23353 raft_consensus.cc:374] T 038e44ad16b74609835844ee46bc48bf P 8322e4d55c1e41cdaa8a33557c47379a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:18.529605 23353 raft_consensus.cc:729] T 038e44ad16b74609835844ee46bc48bf P 8322e4d55c1e41cdaa8a33557c47379a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8322e4d55c1e41cdaa8a33557c47379a, State: Initialized, Role: FOLLOWER
I20260928 06:39:18.529717 23354 consensus_queue.cc:260] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8322e4d55c1e41cdaa8a33557c47379a" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 43941 } } peers { permanent_uuid: "d7068e357a1d49df89eef4fef0908d8c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 36879 } } peers { permanent_uuid: "0b00509ca863446eb188b2595e524b4d" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43677 } }
I20260928 06:39:18.529678 23352 raft_consensus.cc:348] T 038e44ad16b74609835844ee46bc48bf P 0b00509ca863446eb188b2595e524b4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8322e4d55c1e41cdaa8a33557c47379a" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 43941 } } peers { permanent_uuid: "d7068e357a1d49df89eef4fef0908d8c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 36879 } } peers { permanent_uuid: "0b00509ca863446eb188b2595e524b4d" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43677 } }
I20260928 06:39:18.529918 23352 raft_consensus.cc:374] T 038e44ad16b74609835844ee46bc48bf P 0b00509ca863446eb188b2595e524b4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:18.529974 23352 raft_consensus.cc:729] T 038e44ad16b74609835844ee46bc48bf P 0b00509ca863446eb188b2595e524b4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b00509ca863446eb188b2595e524b4d, State: Initialized, Role: FOLLOWER
I20260928 06:39:18.530002 23353 consensus_queue.cc:260] T 038e44ad16b74609835844ee46bc48bf P 8322e4d55c1e41cdaa8a33557c47379a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8322e4d55c1e41cdaa8a33557c47379a" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 43941 } } peers { permanent_uuid: "d7068e357a1d49df89eef4fef0908d8c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 36879 } } peers { permanent_uuid: "0b00509ca863446eb188b2595e524b4d" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43677 } }
I20260928 06:39:18.530433 23352 consensus_queue.cc:260] T 038e44ad16b74609835844ee46bc48bf P 0b00509ca863446eb188b2595e524b4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8322e4d55c1e41cdaa8a33557c47379a" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 43941 } } peers { permanent_uuid: "d7068e357a1d49df89eef4fef0908d8c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 36879 } } peers { permanent_uuid: "0b00509ca863446eb188b2595e524b4d" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43677 } }
I20260928 06:39:18.530544 23335 heartbeater.cc:503] Master 127.5.22.62:41051 was elected leader, sending a full tablet report...
I20260928 06:39:18.530956 23354 ts_tablet_manager.cc:1428] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 06:39:18.530977 23353 ts_tablet_manager.cc:1428] T 038e44ad16b74609835844ee46bc48bf P 8322e4d55c1e41cdaa8a33557c47379a: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260928 06:39:18.531255 23352 ts_tablet_manager.cc:1428] T 038e44ad16b74609835844ee46bc48bf P 0b00509ca863446eb188b2595e524b4d: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260928 06:39:18.531309 23203 heartbeater.cc:503] Master 127.5.22.62:41051 was elected leader, sending a full tablet report...
I20260928 06:39:18.531566 23071 heartbeater.cc:503] Master 127.5.22.62:41051 was elected leader, sending a full tablet report...
W20260928 06:39:18.575778 23072 tablet.cc:2369] T 038e44ad16b74609835844ee46bc48bf P 0b00509ca863446eb188b2595e524b4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:39:18.608789 23358 raft_consensus.cc:482] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:39:18.609007 23358 raft_consensus.cc:504] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8322e4d55c1e41cdaa8a33557c47379a" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 43941 } } peers { permanent_uuid: "d7068e357a1d49df89eef4fef0908d8c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 36879 } } peers { permanent_uuid: "0b00509ca863446eb188b2595e524b4d" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43677 } }
I20260928 06:39:18.610271 23358 leader_election.cc:288] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8322e4d55c1e41cdaa8a33557c47379a (127.5.22.2:43941), 0b00509ca863446eb188b2595e524b4d (127.5.22.1:43677)
I20260928 06:39:18.614832 23347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:18.611005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:49073 (local address 127.5.22.2:43941)
0928 06:39:18.611154 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:18.611159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:18.611172 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:39:18.611403 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:18.611407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:18.611475 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:18.611531 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:18.612022 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.612654 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.613755 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.614175 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.614283 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:18.614368 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:18.614613 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 06:39:18.614687 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 06:39:18.614917 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:18.610609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:43941 (local address 127.5.22.3:49073)
0928 06:39:18.611096 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:18.611111 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:18.611239 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:18.611690 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:18.611695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:18.611716 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:18.611878 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:18.611886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.612769 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.612774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:18.613614 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:18.613624 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.614285 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.614290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:18.614450 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:18.614489 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 06:39:18.614573 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 06:39:18.614670 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":1,"thread_start_us":144,"threads_started":1}
I20260928 06:39:18.615568 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:18.610578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:43677 (local address 127.5.22.3:38091)
0928 06:39:18.610778 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:18.610793 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:18.610919 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:18.611372 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:18.611375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:18.611393 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:18.611510 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:18.611517 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.612439 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.612445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:18.614069 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:18.614084 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.615160 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.615164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:18.615269 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:18.615290 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 06:39:18.615382 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 06:39:18.615456 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":68}
I20260928 06:39:18.615598 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:18.610678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:38091 (local address 127.5.22.1:43677)
0928 06:39:18.610814 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:18.610819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:18.611093 (+ 274us) server_negotiation.cc:407] Connection header received
0928 06:39:18.611164 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:18.611169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:18.611236 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:18.611349 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:18.611631 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.612304 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.614224 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.614794 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.614900 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:18.614960 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:18.615422 (+ 462us) server_negotiation.cc:299] Negotiation successful
0928 06:39:18.615489 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 06:39:18.615742 23157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038e44ad16b74609835844ee46bc48bf" candidate_uuid: "d7068e357a1d49df89eef4fef0908d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8322e4d55c1e41cdaa8a33557c47379a" is_pre_election: true
I20260928 06:39:18.616087 23157 raft_consensus.cc:2454] T 038e44ad16b74609835844ee46bc48bf P 8322e4d55c1e41cdaa8a33557c47379a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7068e357a1d49df89eef4fef0908d8c in term 0.
I20260928 06:39:18.616250 23024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038e44ad16b74609835844ee46bc48bf" candidate_uuid: "d7068e357a1d49df89eef4fef0908d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b00509ca863446eb188b2595e524b4d" is_pre_election: true
I20260928 06:39:18.616572 23024 raft_consensus.cc:2454] T 038e44ad16b74609835844ee46bc48bf P 0b00509ca863446eb188b2595e524b4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7068e357a1d49df89eef4fef0908d8c in term 0.
I20260928 06:39:18.616727 23221 leader_election.cc:302] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8322e4d55c1e41cdaa8a33557c47379a, d7068e357a1d49df89eef4fef0908d8c; no voters:
I20260928 06:39:18.617267 23358 raft_consensus.cc:2781] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:39:18.617368 23358 raft_consensus.cc:482] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:39:18.617422 23358 raft_consensus.cc:3037] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:18.619109 23358 raft_consensus.cc:504] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8322e4d55c1e41cdaa8a33557c47379a" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 43941 } } peers { permanent_uuid: "d7068e357a1d49df89eef4fef0908d8c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 36879 } } peers { permanent_uuid: "0b00509ca863446eb188b2595e524b4d" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43677 } }
I20260928 06:39:18.619656 23358 leader_election.cc:288] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c [CANDIDATE]: Term 1 election: Requested vote from peers 8322e4d55c1e41cdaa8a33557c47379a (127.5.22.2:43941), 0b00509ca863446eb188b2595e524b4d (127.5.22.1:43677)
I20260928 06:39:18.620146 23157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038e44ad16b74609835844ee46bc48bf" candidate_uuid: "d7068e357a1d49df89eef4fef0908d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8322e4d55c1e41cdaa8a33557c47379a"
I20260928 06:39:18.620276 23157 raft_consensus.cc:3037] T 038e44ad16b74609835844ee46bc48bf P 8322e4d55c1e41cdaa8a33557c47379a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:18.621508 23157 raft_consensus.cc:2454] T 038e44ad16b74609835844ee46bc48bf P 8322e4d55c1e41cdaa8a33557c47379a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7068e357a1d49df89eef4fef0908d8c in term 1.
I20260928 06:39:18.621824 23024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038e44ad16b74609835844ee46bc48bf" candidate_uuid: "d7068e357a1d49df89eef4fef0908d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b00509ca863446eb188b2595e524b4d"
I20260928 06:39:18.621887 23221 leader_election.cc:302] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8322e4d55c1e41cdaa8a33557c47379a, d7068e357a1d49df89eef4fef0908d8c; no voters:
I20260928 06:39:18.621989 23024 raft_consensus.cc:3037] T 038e44ad16b74609835844ee46bc48bf P 0b00509ca863446eb188b2595e524b4d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:18.622087 23358 raft_consensus.cc:2781] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:18.622372 23358 raft_consensus.cc:686] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c [term 1 LEADER]: Becoming Leader. State: Replica: d7068e357a1d49df89eef4fef0908d8c, State: Running, Role: LEADER
I20260928 06:39:18.622813 23358 consensus_queue.cc:237] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8322e4d55c1e41cdaa8a33557c47379a" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 43941 } } peers { permanent_uuid: "d7068e357a1d49df89eef4fef0908d8c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 36879 } } peers { permanent_uuid: "0b00509ca863446eb188b2595e524b4d" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43677 } }
I20260928 06:39:18.623359 23024 raft_consensus.cc:2454] T 038e44ad16b74609835844ee46bc48bf P 0b00509ca863446eb188b2595e524b4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7068e357a1d49df89eef4fef0908d8c in term 1.
I20260928 06:39:18.626947 22879 catalog_manager.cc:5495] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c reported cstate change: term changed from 0 to 1, leader changed from <none> to d7068e357a1d49df89eef4fef0908d8c (127.5.22.3). New cstate: current_term: 1 leader_uuid: "d7068e357a1d49df89eef4fef0908d8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8322e4d55c1e41cdaa8a33557c47379a" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 43941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7068e357a1d49df89eef4fef0908d8c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 36879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b00509ca863446eb188b2595e524b4d" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43677 } health_report { overall_health: UNKNOWN } } }
I20260928 06:39:18.660159 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:18.655597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42042 (local address 127.5.22.3:36879)
0928 06:39:18.655747 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:18.655751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:18.655797 (+ 46us) server_negotiation.cc:407] Connection header received
0928 06:39:18.656017 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:18.656020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:18.656075 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:18.656167 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:18.656513 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.657012 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.658583 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:18.658803 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:18.658889 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:18.659290 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:18.659729 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:18.659825 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:18.659946 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 06:39:18.659992 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 06:39:18.666085 23157 raft_consensus.cc:1261] T 038e44ad16b74609835844ee46bc48bf P 8322e4d55c1e41cdaa8a33557c47379a [term 1 FOLLOWER]: Refusing update from remote peer d7068e357a1d49df89eef4fef0908d8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:18.666235 23024 raft_consensus.cc:1261] T 038e44ad16b74609835844ee46bc48bf P 0b00509ca863446eb188b2595e524b4d [term 1 FOLLOWER]: Refusing update from remote peer d7068e357a1d49df89eef4fef0908d8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:18.666877 23358 consensus_queue.cc:1035] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8322e4d55c1e41cdaa8a33557c47379a" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 43941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:18.667048 23362 consensus_queue.cc:1035] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b00509ca863446eb188b2595e524b4d" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 43677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:18.677935 23365 mvcc.cc:205] Tried to move back new op lower bound from 7334205680288395264 to 7334205680122892288. Current Snapshot: MvccSnapshot[applied={T|T < 7334205680288395264}]
I20260928 06:39:18.678363 23367 mvcc.cc:205] Tried to move back new op lower bound from 7334205680288395264 to 7334205680122892288. Current Snapshot: MvccSnapshot[applied={T|T < 7334205680288395264}]
I20260928 06:39:18.677204 23366 mvcc.cc:205] Tried to move back new op lower bound from 7334205680288395264 to 7334205680122892288. Current Snapshot: MvccSnapshot[applied={T|T < 7334205680288395264}]
I20260928 06:39:18.687789 22880 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:39:18.688011 22880 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:39:18.691473 22880 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08cb7721e35a447b8be4397e1124027c: Sending DeleteTablet for 3 replicas of tablet 038e44ad16b74609835844ee46bc48bf
I20260928 06:39:18.692488 23005 tablet_service.cc:1514] Processing DeleteTablet for tablet 038e44ad16b74609835844ee46bc48bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:18 UTC) from {username='slave'} at 127.0.0.1:55390
I20260928 06:39:18.692588 23269 tablet_service.cc:1514] Processing DeleteTablet for tablet 038e44ad16b74609835844ee46bc48bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:18 UTC) from {username='slave'} at 127.0.0.1:42028
I20260928 06:39:18.692646 23137 tablet_service.cc:1514] Processing DeleteTablet for tablet 038e44ad16b74609835844ee46bc48bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:18 UTC) from {username='slave'} at 127.0.0.1:44324
I20260928 06:39:18.693609 23377 tablet_replica.cc:330] T 038e44ad16b74609835844ee46bc48bf P 8322e4d55c1e41cdaa8a33557c47379a: stopping tablet replica
I20260928 06:39:18.693722 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 22942
I20260928 06:39:18.694715 23377 raft_consensus.cc:2229] T 038e44ad16b74609835844ee46bc48bf P 8322e4d55c1e41cdaa8a33557c47379a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:39:18.695147 23377 raft_consensus.cc:2258] T 038e44ad16b74609835844ee46bc48bf P 8322e4d55c1e41cdaa8a33557c47379a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:18.697211 23377 ts_tablet_manager.cc:1905] T 038e44ad16b74609835844ee46bc48bf P 8322e4d55c1e41cdaa8a33557c47379a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:39:18.701215 23377 ts_tablet_manager.cc:1918] T 038e44ad16b74609835844ee46bc48bf P 8322e4d55c1e41cdaa8a33557c47379a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:39:18.701426 23377 log.cc:1196] T 038e44ad16b74609835844ee46bc48bf P 8322e4d55c1e41cdaa8a33557c47379a: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577308481727-5208-0/minicluster-data/ts-1/wal/wals/038e44ad16b74609835844ee46bc48bf
W20260928 06:39:18.701754 22867 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.22.1:43677 (error 108)
I20260928 06:39:18.701977 23377 ts_tablet_manager.cc:1939] T 038e44ad16b74609835844ee46bc48bf P 8322e4d55c1e41cdaa8a33557c47379a: Deleting consensus metadata
I20260928 06:39:18.703207 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 23074
I20260928 06:39:18.703272 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:18.702961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:43677 (local address 127.0.0.1:55396)
0928 06:39:18.703101 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:18.703185 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.1:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 06:39:18.703831 22867 catalog_manager.cc:4584] TS 0b00509ca863446eb188b2595e524b4d (127.5.22.1:43677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 038e44ad16b74609835844ee46bc48bf: Network error: Client connection negotiation failed: client connection to 127.5.22.1:43677: connect: Connection refused (error 111)
I20260928 06:39:18.706595 23375 tablet_replica.cc:330] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c: stopping tablet replica
I20260928 06:39:18.707218 23375 raft_consensus.cc:2229] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:39:18.707845 23375 raft_consensus.cc:2258] T 038e44ad16b74609835844ee46bc48bf P d7068e357a1d49df89eef4fef0908d8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:18.713387 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 23206
I20260928 06:39:18.715842 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:18.715501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:43941 (local address 127.0.0.1:44336)
0928 06:39:18.715660 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:18.715731 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.2:43941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 06:39:18.720355 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 22850
2026-09-28T06:39:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1100 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:18 dist-test-slave-j0qf krb5kdc[23381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:18 dist-test-slave-j0qf krb5kdc[23381](info): set up 2 sockets
Sep 28 06:39:18 dist-test-slave-j0qf krb5kdc[23381](info): commencing operation
krb5kdc: starting...
W20260928 06:39:19.788790 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:39:19 dist-test-slave-j0qf krb5kdc[23381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:39:19.918781 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:38103
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:37631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:38103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:20.032492 23395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:20.032773 23395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:20.032812 23395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:20.036760 23395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:20.036819 23395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:20.036839 23395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:20.036856 23395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:20.036871 23395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:20.042213 23395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:37631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:38103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:38103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:20.043386 23395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:20.044354 23395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:20.053360 23400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:20.053464 23395 server_base.cc:1034] running on GCE node
W20260928 06:39:20.053385 23403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:20.053383 23401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:20.054075 23395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:20.055248 23395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:20.056475 23395 hybrid_clock.cc:648] HybridClock initialized: now 1790577560056447 us; error 38 us; skew 500 ppm
Sep 28 06:39:20 dist-test-slave-j0qf krb5kdc[23381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577560, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:20.063233 23395 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:39:20.064271 23395 webserver.cc:460] Webserver started at http://127.5.22.62:41293/ using document root <none> and password file <none>
I20260928 06:39:20.064868 23395 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:20.064970 23395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:20.065207 23395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:20.067420 23395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "4320bd0109e54981b118e6ec2c55f933"
format_stamp: "Formatted at 2026-09-28 06:39:20 on dist-test-slave-j0qf"
server_key: "e7debe5aa11f9cea965b9513b42264f3"
server_key_iv: "60e6c6cb5959da745ec0eb27883ffb93"
server_key_version: "encryptionkey@0"
I20260928 06:39:20.068001 23395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "4320bd0109e54981b118e6ec2c55f933"
format_stamp: "Formatted at 2026-09-28 06:39:20 on dist-test-slave-j0qf"
server_key: "e7debe5aa11f9cea965b9513b42264f3"
server_key_iv: "60e6c6cb5959da745ec0eb27883ffb93"
server_key_version: "encryptionkey@0"
I20260928 06:39:20.072010 23395 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 06:39:20.074601 23410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:20.075935 23395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:39:20.076105 23395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "4320bd0109e54981b118e6ec2c55f933"
format_stamp: "Formatted at 2026-09-28 06:39:20 on dist-test-slave-j0qf"
server_key: "e7debe5aa11f9cea965b9513b42264f3"
server_key_iv: "60e6c6cb5959da745ec0eb27883ffb93"
server_key_version: "encryptionkey@0"
I20260928 06:39:20.076236 23395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:20.088919 23395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:20.092725 23395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:20.092953 23395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:20.100657 23395 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:38103
I20260928 06:39:20.100687 23462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:38103 every 8 connection(s)
I20260928 06:39:20.101624 23395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:20.104411 23463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:20.105755 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 23395
I20260928 06:39:20.105908 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:20.106237 5208 external_mini_cluster.cc:1428] Setting key cdf494708b35b6c0bc71bf399e084ed9
I20260928 06:39:20.110666 23463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4320bd0109e54981b118e6ec2c55f933: Bootstrap starting.
I20260928 06:39:20.112958 23463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4320bd0109e54981b118e6ec2c55f933: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:20.113683 23463 log.cc:824] T 00000000000000000000000000000000 P 4320bd0109e54981b118e6ec2c55f933: Log is configured to *not* fsync() on all Append() calls
Sep 28 06:39:20 dist-test-slave-j0qf krb5kdc[23381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:39:20.116061 23463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4320bd0109e54981b118e6ec2c55f933: No bootstrap required, opened a new log
I20260928 06:39:20.121227 23463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4320bd0109e54981b118e6ec2c55f933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4320bd0109e54981b118e6ec2c55f933" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38103 } }
I20260928 06:39:20.121510 23463 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4320bd0109e54981b118e6ec2c55f933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:20.121568 23463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4320bd0109e54981b118e6ec2c55f933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4320bd0109e54981b118e6ec2c55f933, State: Initialized, Role: FOLLOWER
I20260928 06:39:20.121724 23466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:20.107793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43580 (local address 127.5.22.62:38103)
0928 06:39:20.108286 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:20.108303 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:20.108341 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:39:20.108988 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:20.109006 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:20.109409 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:20.109795 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:20.110688 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.111732 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.112647 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.113184 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.113378 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:20.116196 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:20.116202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:20.116219 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:20.116252 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:20.118411 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:20.118890 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:20.118896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:20.118902 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:20.119052 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:20.119369 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:20.119372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:20.119374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:20.119914 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:20.120145 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:20.120371 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 06:39:20.120705 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":136,"threads_started":1}
I20260928 06:39:20.122077 23463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4320bd0109e54981b118e6ec2c55f933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4320bd0109e54981b118e6ec2c55f933" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38103 } }
I20260928 06:39:20.122215 23463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4320bd0109e54981b118e6ec2c55f933 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:20.122304 23463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4320bd0109e54981b118e6ec2c55f933 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:20.122466 23463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4320bd0109e54981b118e6ec2c55f933 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:20.123499 23463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4320bd0109e54981b118e6ec2c55f933 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4320bd0109e54981b118e6ec2c55f933" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38103 } }
I20260928 06:39:20.123867 23463 leader_election.cc:302] T 00000000000000000000000000000000 P 4320bd0109e54981b118e6ec2c55f933 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4320bd0109e54981b118e6ec2c55f933; no voters:
I20260928 06:39:20.124229 23463 leader_election.cc:288] T 00000000000000000000000000000000 P 4320bd0109e54981b118e6ec2c55f933 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:20.124346 23468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4320bd0109e54981b118e6ec2c55f933 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:20.124536 23468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4320bd0109e54981b118e6ec2c55f933 [term 1 LEADER]: Becoming Leader. State: Replica: 4320bd0109e54981b118e6ec2c55f933, State: Running, Role: LEADER
I20260928 06:39:20.124838 23468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4320bd0109e54981b118e6ec2c55f933 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4320bd0109e54981b118e6ec2c55f933" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38103 } }
I20260928 06:39:20.125626 23463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4320bd0109e54981b118e6ec2c55f933 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:20.126737 23469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4320bd0109e54981b118e6ec2c55f933 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4320bd0109e54981b118e6ec2c55f933. Latest consensus state: current_term: 1 leader_uuid: "4320bd0109e54981b118e6ec2c55f933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4320bd0109e54981b118e6ec2c55f933" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38103 } } }
I20260928 06:39:20.126883 23469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4320bd0109e54981b118e6ec2c55f933 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:20.127319 23477 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:20.127336 23470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4320bd0109e54981b118e6ec2c55f933 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4320bd0109e54981b118e6ec2c55f933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4320bd0109e54981b118e6ec2c55f933" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 38103 } } }
I20260928 06:39:20.127497 23470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4320bd0109e54981b118e6ec2c55f933 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:20.130846 23477 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:20.136766 23477 catalog_manager.cc:1347] Generated new cluster ID: 48c9c865ab344237a7ae1568cdb03e6a
I20260928 06:39:20.136874 23477 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:20.160609 23477 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:20.161590 23477 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:20.176990 23477 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4320bd0109e54981b118e6ec2c55f933: Generated new TSK 0
I20260928 06:39:20.177917 23477 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.22.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
I20260928 06:39:20.227186 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:38103
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:37631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:20.352617 23489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:20.352881 23489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:20.352924 23489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:20.357127 23489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:20.357193 23489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:20.357251 23489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:39:20.362474 23489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:37631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:38103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:20.363461 23489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:20.364518 23489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:20.375037 23495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:20.375016 23497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:20.375016 23494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:20.375057 23489 server_base.cc:1034] running on GCE node
I20260928 06:39:20.375826 23489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:20.376508 23489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:20.377710 23489 hybrid_clock.cc:648] HybridClock initialized: now 1790577560377697 us; error 38 us; skew 500 ppm
Sep 28 06:39:20 dist-test-slave-j0qf krb5kdc[23381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577560, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:20.380818 23489 init.cc:377] Logged in from keytab as kudu/127.5.22.1@KRBTEST.COM (short username kudu)
I20260928 06:39:20.381800 23489 webserver.cc:460] Webserver started at http://127.5.22.1:45453/ using document root <none> and password file <none>
I20260928 06:39:20.382459 23489 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:20.382562 23489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:20.382793 23489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:20.385004 23489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "60a44cf0ecd24cf4af25477f0fe8dda6"
format_stamp: "Formatted at 2026-09-28 06:39:20 on dist-test-slave-j0qf"
server_key: "26c299784e54b46a1afcabcc47717c17"
server_key_iv: "b03e8d1e2f81b62ee2f538ed26087031"
server_key_version: "encryptionkey@0"
I20260928 06:39:20.385593 23489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "60a44cf0ecd24cf4af25477f0fe8dda6"
format_stamp: "Formatted at 2026-09-28 06:39:20 on dist-test-slave-j0qf"
server_key: "26c299784e54b46a1afcabcc47717c17"
server_key_iv: "b03e8d1e2f81b62ee2f538ed26087031"
server_key_version: "encryptionkey@0"
I20260928 06:39:20.389586 23489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 06:39:20.392277 23504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:20.393603 23489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:39:20.393755 23489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "60a44cf0ecd24cf4af25477f0fe8dda6"
format_stamp: "Formatted at 2026-09-28 06:39:20 on dist-test-slave-j0qf"
server_key: "26c299784e54b46a1afcabcc47717c17"
server_key_iv: "b03e8d1e2f81b62ee2f538ed26087031"
server_key_version: "encryptionkey@0"
I20260928 06:39:20.393884 23489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:20.410197 23489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:20.413554 23489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:20.413761 23489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:20.414435 23489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:20.415344 23489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:20.415419 23489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:20.415491 23489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:20.415537 23489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:20.424515 23489 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:42579
I20260928 06:39:20.424541 23619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:42579 every 8 connection(s)
I20260928 06:39:20.425436 23489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:39:20 dist-test-slave-j0qf krb5kdc[23381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577560, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:39:20.435390 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 23489
I20260928 06:39:20.435554 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:39:20.435925 5208 external_mini_cluster.cc:1428] Setting key 0ce8b352647e9e4030d681e66d5b563d
I20260928 06:39:20.439549 23466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:20.427439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:33169 (local address 127.5.22.62:38103)
0928 06:39:20.427574 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:20.427579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:20.428562 (+ 983us) server_negotiation.cc:407] Connection header received
0928 06:39:20.429395 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:20.429398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:20.429444 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:20.429521 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:20.430712 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.431245 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.432514 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.432733 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.432812 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:20.435554 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:20.435558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:20.435565 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:20.435599 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:20.437372 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:20.437949 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:20.437955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:20.437958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:20.438042 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:20.438435 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:20.438439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:20.438441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:20.438626 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:20.438740 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:20.439264 (+ 524us) server_negotiation.cc:299] Negotiation successful
0928 06:39:20.439391 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:39:20.440415 23622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:20.427927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38103 (local address 127.5.22.1:33169)
0928 06:39:20.428397 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:20.428441 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:20.429184 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:20.429692 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:20.429701 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:20.430040 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:20.430578 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:20.430593 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.431383 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.431386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:20.432396 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:20.432406 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.432868 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.432871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:20.433010 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:20.433661 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:20.433683 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:20.435354 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:20.437555 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:20.437560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:20.437572 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:20.437833 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:20.438143 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:20.438147 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:20.438148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:20.438300 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:20.438739 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:20.438745 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:20.439012 (+ 267us) client_negotiation.cc:769] Sending connection context
0928 06:39:20.439209 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 06:39:20.439570 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":112,"threads_started":1}
I20260928 06:39:20.441735 23620 heartbeater.cc:347] Connected to a master server at 127.5.22.62:38103
I20260928 06:39:20.442021 23620 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:20.442682 23620 heartbeater.cc:511] Master 127.5.22.62:38103 requested a full tablet report, sending...
I20260928 06:39:20.444553 23427 ts_manager.cc:194] Registered new tserver with Master: 60a44cf0ecd24cf4af25477f0fe8dda6 (127.5.22.1:42579)
I20260928 06:39:20.446234 23427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.1@KRBTEST.COM'} at 127.5.22.1:33169
WARNING: no policy specified for kudu/127.5.22.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
I20260928 06:39:20.467127 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:38103
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:37631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:20.580034 23625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:20.580372 23625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:20.580420 23625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:20.584473 23625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:20.584525 23625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:20.584621 23625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:39:20.589867 23625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:37631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:38103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:20.590998 23625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:20.591974 23625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:20.602448 23633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:20.602442 23631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:20.602490 23625 server_base.cc:1034] running on GCE node
W20260928 06:39:20.602444 23630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:20.603256 23625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:20.603909 23625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:20.605083 23625 hybrid_clock.cc:648] HybridClock initialized: now 1790577560605045 us; error 51 us; skew 500 ppm
Sep 28 06:39:20 dist-test-slave-j0qf krb5kdc[23381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577560, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:20.607980 23625 init.cc:377] Logged in from keytab as kudu/127.5.22.2@KRBTEST.COM (short username kudu)
I20260928 06:39:20.608973 23625 webserver.cc:460] Webserver started at http://127.5.22.2:34087/ using document root <none> and password file <none>
I20260928 06:39:20.609582 23625 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:20.609784 23625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:20.610037 23625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:20.612187 23625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "b6a469f7238346c889533948f9298c31"
format_stamp: "Formatted at 2026-09-28 06:39:20 on dist-test-slave-j0qf"
server_key: "924ba44b765e765a3c4ea3b53320180a"
server_key_iv: "6b1f579d1718e2f4c4d1476866a0c6d8"
server_key_version: "encryptionkey@0"
I20260928 06:39:20.612771 23625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "b6a469f7238346c889533948f9298c31"
format_stamp: "Formatted at 2026-09-28 06:39:20 on dist-test-slave-j0qf"
server_key: "924ba44b765e765a3c4ea3b53320180a"
server_key_iv: "6b1f579d1718e2f4c4d1476866a0c6d8"
server_key_version: "encryptionkey@0"
I20260928 06:39:20.616526 23625 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 06:39:20.619001 23640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:20.620258 23625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:39:20.620424 23625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "b6a469f7238346c889533948f9298c31"
format_stamp: "Formatted at 2026-09-28 06:39:20 on dist-test-slave-j0qf"
server_key: "924ba44b765e765a3c4ea3b53320180a"
server_key_iv: "6b1f579d1718e2f4c4d1476866a0c6d8"
server_key_version: "encryptionkey@0"
I20260928 06:39:20.620549 23625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:20.637571 23625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:20.641377 23625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:20.641611 23625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:20.642272 23625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:20.643191 23625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:20.643270 23625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:20.643342 23625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:20.643402 23625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:20.652971 23625 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:34235
I20260928 06:39:20.652994 23755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:34235 every 8 connection(s)
I20260928 06:39:20.653914 23625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:39:20 dist-test-slave-j0qf krb5kdc[23381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577560, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:39:20.664258 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 23625
I20260928 06:39:20.664398 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:39:20.664708 5208 external_mini_cluster.cc:1428] Setting key b8618e615c745c701664899f190a3220
I20260928 06:39:20.668843 23466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:20.656213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:53181 (local address 127.5.22.62:38103)
0928 06:39:20.656371 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:20.656375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:20.657358 (+ 983us) server_negotiation.cc:407] Connection header received
0928 06:39:20.658281 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:20.658285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:20.658365 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:20.658452 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:20.659699 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.660235 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.661569 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.661786 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.661874 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:20.664948 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:20.664952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:20.664955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:20.665005 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:20.666696 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:20.667280 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:20.667285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:20.667287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:20.667370 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:20.667738 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:20.667741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:20.667743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:20.667927 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:20.668043 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:20.668550 (+ 507us) server_negotiation.cc:299] Negotiation successful
0928 06:39:20.668678 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 06:39:20.669974 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:20.656629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38103 (local address 127.5.22.2:53181)
0928 06:39:20.657149 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:20.657195 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:20.658058 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:20.658624 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:20.658632 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:20.659024 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:20.659548 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:20.659564 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.660364 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.660367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:20.661442 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:20.661453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.661957 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.661960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:20.662107 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:20.662890 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:20.662907 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:20.664747 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:20.666874 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:20.666880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:20.666902 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:20.667160 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:20.667469 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:20.667473 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:20.667475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:20.667626 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:20.668041 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:20.668048 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:20.668304 (+ 256us) client_negotiation.cc:769] Sending connection context
0928 06:39:20.668510 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 06:39:20.668851 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":134,"threads_started":1}
I20260928 06:39:20.671511 23756 heartbeater.cc:347] Connected to a master server at 127.5.22.62:38103
I20260928 06:39:20.671792 23756 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:20.672359 23756 heartbeater.cc:511] Master 127.5.22.62:38103 requested a full tablet report, sending...
I20260928 06:39:20.673561 23427 ts_manager.cc:194] Registered new tserver with Master: b6a469f7238346c889533948f9298c31 (127.5.22.2:34235)
I20260928 06:39:20.674219 23427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.2@KRBTEST.COM'} at 127.5.22.2:53181
WARNING: no policy specified for kudu/127.5.22.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
I20260928 06:39:20.697548 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:38103
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:37631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:20.814633 23761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:20.814977 23761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:20.815058 23761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:20.819159 23761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:20.819286 23761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:20.819391 23761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:39:20.825388 23761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:37631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:38103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:20.826653 23761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:20.828058 23761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:20.838406 23769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:20.838431 23767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:20.838406 23766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:20.838698 23761 server_base.cc:1034] running on GCE node
I20260928 06:39:20.839184 23761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:20.839846 23761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:20.841008 23761 hybrid_clock.cc:648] HybridClock initialized: now 1790577560840936 us; error 87 us; skew 500 ppm
Sep 28 06:39:20 dist-test-slave-j0qf krb5kdc[23381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577560, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:20.843942 23761 init.cc:377] Logged in from keytab as kudu/127.5.22.3@KRBTEST.COM (short username kudu)
I20260928 06:39:20.844882 23761 webserver.cc:460] Webserver started at http://127.5.22.3:44589/ using document root <none> and password file <none>
I20260928 06:39:20.845507 23761 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:20.845592 23761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:20.845821 23761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:20.847971 23761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "3e62ef28eb534f2d88b5f4c4aa3af3e7"
format_stamp: "Formatted at 2026-09-28 06:39:20 on dist-test-slave-j0qf"
server_key: "95f0d97e45b99dd8529efe105cf416c1"
server_key_iv: "f10da624bd9152ee25d12f198923f255"
server_key_version: "encryptionkey@0"
I20260928 06:39:20.848547 23761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "3e62ef28eb534f2d88b5f4c4aa3af3e7"
format_stamp: "Formatted at 2026-09-28 06:39:20 on dist-test-slave-j0qf"
server_key: "95f0d97e45b99dd8529efe105cf416c1"
server_key_iv: "f10da624bd9152ee25d12f198923f255"
server_key_version: "encryptionkey@0"
I20260928 06:39:20.852535 23761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:39:20.855126 23776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:20.856456 23761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:39:20.856554 23761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "3e62ef28eb534f2d88b5f4c4aa3af3e7"
format_stamp: "Formatted at 2026-09-28 06:39:20 on dist-test-slave-j0qf"
server_key: "95f0d97e45b99dd8529efe105cf416c1"
server_key_iv: "f10da624bd9152ee25d12f198923f255"
server_key_version: "encryptionkey@0"
I20260928 06:39:20.856701 23761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:20.870308 23761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:20.873399 23761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:20.873605 23761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:20.874256 23761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:20.875151 23761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:20.875203 23761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:20.875242 23761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:20.875286 23761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:20.885931 23761 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:42583
I20260928 06:39:20.885980 23890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:42583 every 8 connection(s)
I20260928 06:39:20.886947 23761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:39:20.894810 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 23761
I20260928 06:39:20.894966 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
I20260928 06:39:20.895310 5208 external_mini_cluster.cc:1428] Setting key bfdaf3546f93b7f278b4d43a76de3ceb
Sep 28 06:39:20 dist-test-slave-j0qf krb5kdc[23381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577560, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:39:20.900888 23466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:20.888896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:53571 (local address 127.5.22.62:38103)
0928 06:39:20.889052 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:20.889057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:20.890148 (+ 1091us) server_negotiation.cc:407] Connection header received
0928 06:39:20.891057 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:20.891060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:20.891106 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:20.891180 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:20.892414 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.892937 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.894132 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.894406 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.894486 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:20.897209 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:20.897213 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:20.897216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:20.897240 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:20.898815 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:20.899357 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:20.899361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:20.899363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:20.899444 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:20.899797 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:20.899801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:20.899803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:20.899990 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:20.900101 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:20.900600 (+ 499us) server_negotiation.cc:299] Negotiation successful
0928 06:39:20.900719 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 06:39:20.901786 23893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:20.889334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:38103 (local address 127.5.22.3:53571)
0928 06:39:20.889996 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:20.890036 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:20.890875 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:20.891392 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:20.891398 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:20.891762 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:20.892265 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:20.892279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.893064 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.893066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:20.894009 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:20.894018 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.894563 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.894568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:20.894770 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:20.895385 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:20.895407 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:20.897008 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:20.898956 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:20.898961 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:20.898974 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:20.899233 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:20.899540 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:20.899544 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:20.899546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:20.899689 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:20.900100 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:20.900107 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:20.900352 (+ 245us) client_negotiation.cc:769] Sending connection context
0928 06:39:20.900545 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 06:39:20.900876 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":111,"threads_started":1}
I20260928 06:39:20.903612 23891 heartbeater.cc:347] Connected to a master server at 127.5.22.62:38103
I20260928 06:39:20.903934 23891 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:20.904500 23891 heartbeater.cc:511] Master 127.5.22.62:38103 requested a full tablet report, sending...
I20260928 06:39:20.906028 23427 ts_manager.cc:194] Registered new tserver with Master: 3e62ef28eb534f2d88b5f4c4aa3af3e7 (127.5.22.3:42583)
I20260928 06:39:20.906739 23427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.3@KRBTEST.COM'} at 127.5.22.3:53571
I20260928 06:39:20.910040 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:39:20.921300 23466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:20.912600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43590 (local address 127.5.22.62:38103)
0928 06:39:20.912883 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:20.912888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:20.912998 (+ 110us) server_negotiation.cc:407] Connection header received
0928 06:39:20.913173 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:20.913178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:20.913229 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:20.913325 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:20.914080 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.914727 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.915619 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.915855 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.915975 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:20.917685 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:20.917689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:20.917692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:20.917724 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:20.919349 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:20.919901 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:20.919907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:20.919911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:20.919993 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:20.920341 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:20.920345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:20.920347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:20.920580 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:20.920750 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:20.921037 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 06:39:20.921133 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260928 06:39:20.924935 23427 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:39:20.928614 23427 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:39:20.945200 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:20.938848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47824 (local address 127.5.22.2:34235)
0928 06:39:20.939359 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:20.939384 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:20.939409 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:39:20.939490 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:20.939496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:20.939741 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:20.939915 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:20.940431 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.941601 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.943322 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.944529 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.944648 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:20.944809 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:20.944981 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 06:39:20.945066 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":167,"threads_started":1}
I20260928 06:39:20.945217 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:20.938665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:34235 (local address 127.0.0.1:47824)
0928 06:39:20.939068 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:20.939128 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:20.939277 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:20.940082 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:20.940086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:20.940105 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:20.940300 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:20.940308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.941758 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.941761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:20.942927 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:20.942937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.944640 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.944643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:20.944770 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:20.944818 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 06:39:20.944931 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 06:39:20.945054 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":1,"thread_start_us":126,"threads_started":1}
I20260928 06:39:20.945827 23904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:20.938941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:42583 (local address 127.0.0.1:55268)
0928 06:39:20.939820 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:20.939836 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:20.939962 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:20.940881 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:20.940884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:20.940902 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:20.941003 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:20.941008 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.942454 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.942458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:20.943619 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:20.943628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.945428 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.945431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:20.945528 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:20.945555 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 06:39:20.945629 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 06:39:20.945701 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":93,"threads_started":1}
I20260928 06:39:20.946273 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:20.939497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:42579 (local address 127.0.0.1:47588)
0928 06:39:20.940518 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:20.940531 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:20.940609 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:20.941865 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:20.941868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:20.941886 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:20.941968 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:20.941973 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.943369 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.943372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:20.944395 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:20.944404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.945888 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.945893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:20.945998 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:20.946022 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 06:39:20.946087 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:39:20.946150 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"mutex_wait_us":3,"thread_start_us":105,"threads_started":1}
I20260928 06:39:20.946357 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:20.939581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55268 (local address 127.5.22.3:42583)
0928 06:39:20.940179 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:20.940191 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:20.940216 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:39:20.940295 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:20.940301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:20.940493 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:20.940652 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:20.941128 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.942251 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.943754 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.944529 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.944648 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:20.944809 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:20.946056 (+ 1247us) server_negotiation.cc:299] Negotiation successful
0928 06:39:20.946162 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":95,"threads_started":1}
I20260928 06:39:20.946821 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:20.939723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47588 (local address 127.5.22.1:42579)
0928 06:39:20.941408 (+ 1685us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:20.941423 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:20.941444 (+ 21us) server_negotiation.cc:407] Connection header received
0928 06:39:20.941514 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:20.941519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:20.941708 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:20.941881 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:20.942086 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.943228 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.944530 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:20.945588 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:20.945680 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:20.945813 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:20.946556 (+ 743us) server_negotiation.cc:299] Negotiation successful
0928 06:39:20.946688 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1568,"thread_start_us":127,"threads_started":1}
I20260928 06:39:20.948549 23689 tablet_service.cc:1467] Processing CreateTablet for tablet 22415ebf846546289160d8b1e5d5d57e (DEFAULT_TABLE table=test-table [id=efb09215095b494c8afb9d5e9834947d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:20.948728 23824 tablet_service.cc:1467] Processing CreateTablet for tablet 22415ebf846546289160d8b1e5d5d57e (DEFAULT_TABLE table=test-table [id=efb09215095b494c8afb9d5e9834947d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:20.949674 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22415ebf846546289160d8b1e5d5d57e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:20.949717 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22415ebf846546289160d8b1e5d5d57e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:20.952107 23552 tablet_service.cc:1467] Processing CreateTablet for tablet 22415ebf846546289160d8b1e5d5d57e (DEFAULT_TABLE table=test-table [id=efb09215095b494c8afb9d5e9834947d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:20.953323 23552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22415ebf846546289160d8b1e5d5d57e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:20.955901 23908 tablet_bootstrap.cc:492] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7: Bootstrap starting.
I20260928 06:39:20.957849 23909 tablet_bootstrap.cc:492] T 22415ebf846546289160d8b1e5d5d57e P b6a469f7238346c889533948f9298c31: Bootstrap starting.
I20260928 06:39:20.958087 23908 tablet_bootstrap.cc:654] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:20.959252 23908 log.cc:824] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:20.960429 23909 tablet_bootstrap.cc:654] T 22415ebf846546289160d8b1e5d5d57e P b6a469f7238346c889533948f9298c31: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:20.961789 23908 tablet_bootstrap.cc:492] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7: No bootstrap required, opened a new log
I20260928 06:39:20.961835 23909 log.cc:824] T 22415ebf846546289160d8b1e5d5d57e P b6a469f7238346c889533948f9298c31: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:20.962018 23908 ts_tablet_manager.cc:1397] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 06:39:20.964046 23909 tablet_bootstrap.cc:492] T 22415ebf846546289160d8b1e5d5d57e P b6a469f7238346c889533948f9298c31: No bootstrap required, opened a new log
I20260928 06:39:20.964267 23909 ts_tablet_manager.cc:1397] T 22415ebf846546289160d8b1e5d5d57e P b6a469f7238346c889533948f9298c31: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 06:39:20.965490 23908 raft_consensus.cc:348] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a469f7238346c889533948f9298c31" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34235 } } peers { permanent_uuid: "60a44cf0ecd24cf4af25477f0fe8dda6" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42579 } } peers { permanent_uuid: "3e62ef28eb534f2d88b5f4c4aa3af3e7" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 42583 } }
I20260928 06:39:20.965764 23908 raft_consensus.cc:374] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:20.965823 23908 raft_consensus.cc:729] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e62ef28eb534f2d88b5f4c4aa3af3e7, State: Initialized, Role: FOLLOWER
I20260928 06:39:20.966301 23908 consensus_queue.cc:260] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a469f7238346c889533948f9298c31" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34235 } } peers { permanent_uuid: "60a44cf0ecd24cf4af25477f0fe8dda6" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42579 } } peers { permanent_uuid: "3e62ef28eb534f2d88b5f4c4aa3af3e7" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 42583 } }
I20260928 06:39:20.967770 23908 ts_tablet_manager.cc:1428] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260928 06:39:20.967958 23909 raft_consensus.cc:348] T 22415ebf846546289160d8b1e5d5d57e P b6a469f7238346c889533948f9298c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a469f7238346c889533948f9298c31" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34235 } } peers { permanent_uuid: "60a44cf0ecd24cf4af25477f0fe8dda6" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42579 } } peers { permanent_uuid: "3e62ef28eb534f2d88b5f4c4aa3af3e7" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 42583 } }
I20260928 06:39:20.968271 23909 raft_consensus.cc:374] T 22415ebf846546289160d8b1e5d5d57e P b6a469f7238346c889533948f9298c31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:20.968288 23891 heartbeater.cc:503] Master 127.5.22.62:38103 was elected leader, sending a full tablet report...
I20260928 06:39:20.968344 23909 raft_consensus.cc:729] T 22415ebf846546289160d8b1e5d5d57e P b6a469f7238346c889533948f9298c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6a469f7238346c889533948f9298c31, State: Initialized, Role: FOLLOWER
I20260928 06:39:20.968451 23912 tablet_bootstrap.cc:492] T 22415ebf846546289160d8b1e5d5d57e P 60a44cf0ecd24cf4af25477f0fe8dda6: Bootstrap starting.
I20260928 06:39:20.968847 23909 consensus_queue.cc:260] T 22415ebf846546289160d8b1e5d5d57e P b6a469f7238346c889533948f9298c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a469f7238346c889533948f9298c31" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34235 } } peers { permanent_uuid: "60a44cf0ecd24cf4af25477f0fe8dda6" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42579 } } peers { permanent_uuid: "3e62ef28eb534f2d88b5f4c4aa3af3e7" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 42583 } }
I20260928 06:39:20.969870 23909 ts_tablet_manager.cc:1428] T 22415ebf846546289160d8b1e5d5d57e P b6a469f7238346c889533948f9298c31: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 06:39:20.970223 23756 heartbeater.cc:503] Master 127.5.22.62:38103 was elected leader, sending a full tablet report...
I20260928 06:39:20.971369 23912 tablet_bootstrap.cc:654] T 22415ebf846546289160d8b1e5d5d57e P 60a44cf0ecd24cf4af25477f0fe8dda6: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:20.972321 23912 log.cc:824] T 22415ebf846546289160d8b1e5d5d57e P 60a44cf0ecd24cf4af25477f0fe8dda6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:20.974766 23912 tablet_bootstrap.cc:492] T 22415ebf846546289160d8b1e5d5d57e P 60a44cf0ecd24cf4af25477f0fe8dda6: No bootstrap required, opened a new log
I20260928 06:39:20.974972 23912 ts_tablet_manager.cc:1397] T 22415ebf846546289160d8b1e5d5d57e P 60a44cf0ecd24cf4af25477f0fe8dda6: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260928 06:39:20.978452 23912 raft_consensus.cc:348] T 22415ebf846546289160d8b1e5d5d57e P 60a44cf0ecd24cf4af25477f0fe8dda6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a469f7238346c889533948f9298c31" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34235 } } peers { permanent_uuid: "60a44cf0ecd24cf4af25477f0fe8dda6" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42579 } } peers { permanent_uuid: "3e62ef28eb534f2d88b5f4c4aa3af3e7" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 42583 } }
I20260928 06:39:20.978726 23912 raft_consensus.cc:374] T 22415ebf846546289160d8b1e5d5d57e P 60a44cf0ecd24cf4af25477f0fe8dda6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:20.978783 23912 raft_consensus.cc:729] T 22415ebf846546289160d8b1e5d5d57e P 60a44cf0ecd24cf4af25477f0fe8dda6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60a44cf0ecd24cf4af25477f0fe8dda6, State: Initialized, Role: FOLLOWER
I20260928 06:39:20.979202 23912 consensus_queue.cc:260] T 22415ebf846546289160d8b1e5d5d57e P 60a44cf0ecd24cf4af25477f0fe8dda6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a469f7238346c889533948f9298c31" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34235 } } peers { permanent_uuid: "60a44cf0ecd24cf4af25477f0fe8dda6" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42579 } } peers { permanent_uuid: "3e62ef28eb534f2d88b5f4c4aa3af3e7" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 42583 } }
I20260928 06:39:20.979997 23620 heartbeater.cc:503] Master 127.5.22.62:38103 was elected leader, sending a full tablet report...
I20260928 06:39:20.980151 23912 ts_tablet_manager.cc:1428] T 22415ebf846546289160d8b1e5d5d57e P 60a44cf0ecd24cf4af25477f0fe8dda6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 06:39:21.010782 23913 raft_consensus.cc:482] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:39:21.011075 23913 raft_consensus.cc:504] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a469f7238346c889533948f9298c31" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34235 } } peers { permanent_uuid: "60a44cf0ecd24cf4af25477f0fe8dda6" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42579 } } peers { permanent_uuid: "3e62ef28eb534f2d88b5f4c4aa3af3e7" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 42583 } }
I20260928 06:39:21.012255 23913 leader_election.cc:288] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6a469f7238346c889533948f9298c31 (127.5.22.2:34235), 60a44cf0ecd24cf4af25477f0fe8dda6 (127.5.22.1:42579)
I20260928 06:39:21.017037 23893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:21.012627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:34235 (local address 127.5.22.3:60995)
0928 06:39:21.012770 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:21.012820 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:21.012915 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:21.013335 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:21.013342 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:21.013361 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:21.013504 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:21.013513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:21.014519 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:21.014523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:21.015766 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:21.015777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:21.016559 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:21.016564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:21.016683 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:21.016712 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 06:39:21.016809 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 06:39:21.016888 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":28}
I20260928 06:39:21.017093 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:21.012816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:60995 (local address 127.5.22.2:34235)
0928 06:39:21.012978 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:21.012983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:21.013001 (+ 18us) server_negotiation.cc:407] Connection header received
0928 06:39:21.013063 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:21.013066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:21.013123 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:21.013219 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:21.013654 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:21.014397 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:21.015928 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:21.016419 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:21.016502 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:21.016556 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:21.016871 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 06:39:21.016956 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 06:39:21.017339 23917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:21.012955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:42579 (local address 127.5.22.3:38639)
0928 06:39:21.013375 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:21.013387 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:21.013470 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:21.013734 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:21.013736 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:21.013780 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:21.013901 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:21.013907 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:21.015050 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:21.015052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:21.015961 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:21.015972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:21.016881 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:21.016885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:21.016987 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:21.017021 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 06:39:21.017108 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:39:21.017187 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":1,"thread_start_us":182,"threads_started":1}
I20260928 06:39:21.017439 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:21.013108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:38639 (local address 127.5.22.1:42579)
0928 06:39:21.013273 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:21.013279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:21.013470 (+ 191us) server_negotiation.cc:407] Connection header received
0928 06:39:21.013557 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:21.013562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:21.013617 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:21.013684 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:21.014064 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:21.014893 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:21.016101 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:21.016710 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:21.016835 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:21.016895 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:21.017166 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 06:39:21.017267 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 06:39:21.017835 23709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22415ebf846546289160d8b1e5d5d57e" candidate_uuid: "3e62ef28eb534f2d88b5f4c4aa3af3e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6a469f7238346c889533948f9298c31" is_pre_election: true
I20260928 06:39:21.018196 23709 raft_consensus.cc:2454] T 22415ebf846546289160d8b1e5d5d57e P b6a469f7238346c889533948f9298c31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e62ef28eb534f2d88b5f4c4aa3af3e7 in term 0.
I20260928 06:39:21.018240 23573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22415ebf846546289160d8b1e5d5d57e" candidate_uuid: "3e62ef28eb534f2d88b5f4c4aa3af3e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60a44cf0ecd24cf4af25477f0fe8dda6" is_pre_election: true
I20260928 06:39:21.018585 23573 raft_consensus.cc:2454] T 22415ebf846546289160d8b1e5d5d57e P 60a44cf0ecd24cf4af25477f0fe8dda6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e62ef28eb534f2d88b5f4c4aa3af3e7 in term 0.
I20260928 06:39:21.018836 23777 leader_election.cc:302] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e62ef28eb534f2d88b5f4c4aa3af3e7, b6a469f7238346c889533948f9298c31; no voters:
I20260928 06:39:21.019116 23913 raft_consensus.cc:2781] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:39:21.019220 23913 raft_consensus.cc:482] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:39:21.019284 23913 raft_consensus.cc:3037] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:21.020339 23913 raft_consensus.cc:504] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a469f7238346c889533948f9298c31" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34235 } } peers { permanent_uuid: "60a44cf0ecd24cf4af25477f0fe8dda6" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42579 } } peers { permanent_uuid: "3e62ef28eb534f2d88b5f4c4aa3af3e7" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 42583 } }
I20260928 06:39:21.020695 23913 leader_election.cc:288] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7 [CANDIDATE]: Term 1 election: Requested vote from peers b6a469f7238346c889533948f9298c31 (127.5.22.2:34235), 60a44cf0ecd24cf4af25477f0fe8dda6 (127.5.22.1:42579)
I20260928 06:39:21.021421 23573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22415ebf846546289160d8b1e5d5d57e" candidate_uuid: "3e62ef28eb534f2d88b5f4c4aa3af3e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60a44cf0ecd24cf4af25477f0fe8dda6"
I20260928 06:39:21.021445 23709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22415ebf846546289160d8b1e5d5d57e" candidate_uuid: "3e62ef28eb534f2d88b5f4c4aa3af3e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6a469f7238346c889533948f9298c31"
I20260928 06:39:21.021571 23709 raft_consensus.cc:3037] T 22415ebf846546289160d8b1e5d5d57e P b6a469f7238346c889533948f9298c31 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:21.021571 23573 raft_consensus.cc:3037] T 22415ebf846546289160d8b1e5d5d57e P 60a44cf0ecd24cf4af25477f0fe8dda6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:21.023077 23573 raft_consensus.cc:2454] T 22415ebf846546289160d8b1e5d5d57e P 60a44cf0ecd24cf4af25477f0fe8dda6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e62ef28eb534f2d88b5f4c4aa3af3e7 in term 1.
I20260928 06:39:21.023077 23709 raft_consensus.cc:2454] T 22415ebf846546289160d8b1e5d5d57e P b6a469f7238346c889533948f9298c31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e62ef28eb534f2d88b5f4c4aa3af3e7 in term 1.
I20260928 06:39:21.023419 23777 leader_election.cc:302] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e62ef28eb534f2d88b5f4c4aa3af3e7, b6a469f7238346c889533948f9298c31; no voters:
I20260928 06:39:21.023640 23913 raft_consensus.cc:2781] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:21.023900 23913 raft_consensus.cc:686] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7 [term 1 LEADER]: Becoming Leader. State: Replica: 3e62ef28eb534f2d88b5f4c4aa3af3e7, State: Running, Role: LEADER
I20260928 06:39:21.024245 23913 consensus_queue.cc:237] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a469f7238346c889533948f9298c31" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34235 } } peers { permanent_uuid: "60a44cf0ecd24cf4af25477f0fe8dda6" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42579 } } peers { permanent_uuid: "3e62ef28eb534f2d88b5f4c4aa3af3e7" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 42583 } }
I20260928 06:39:21.028034 23426 catalog_manager.cc:5495] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e62ef28eb534f2d88b5f4c4aa3af3e7 (127.5.22.3). New cstate: current_term: 1 leader_uuid: "3e62ef28eb534f2d88b5f4c4aa3af3e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a469f7238346c889533948f9298c31" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60a44cf0ecd24cf4af25477f0fe8dda6" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e62ef28eb534f2d88b5f4c4aa3af3e7" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 42583 } health_report { overall_health: HEALTHY } } }
I20260928 06:39:21.051518 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:21.047817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55270 (local address 127.5.22.3:42583)
0928 06:39:21.047982 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:21.047989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:21.048008 (+ 19us) server_negotiation.cc:407] Connection header received
0928 06:39:21.048096 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:21.048099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:21.048153 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:21.048272 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:21.048559 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:21.049069 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:21.049957 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:21.050185 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:21.050305 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:21.050650 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:21.051052 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:21.051225 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:21.051311 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 06:39:21.051351 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 06:39:21.057507 23573 raft_consensus.cc:1261] T 22415ebf846546289160d8b1e5d5d57e P 60a44cf0ecd24cf4af25477f0fe8dda6 [term 1 FOLLOWER]: Refusing update from remote peer 3e62ef28eb534f2d88b5f4c4aa3af3e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:21.057539 23709 raft_consensus.cc:1261] T 22415ebf846546289160d8b1e5d5d57e P b6a469f7238346c889533948f9298c31 [term 1 FOLLOWER]: Refusing update from remote peer 3e62ef28eb534f2d88b5f4c4aa3af3e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:21.058269 23913 consensus_queue.cc:1035] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60a44cf0ecd24cf4af25477f0fe8dda6" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:21.058523 23918 consensus_queue.cc:1035] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a469f7238346c889533948f9298c31" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:21.069021 23923 mvcc.cc:205] Tried to move back new op lower bound from 7334205690083942400 to 7334205689958260736. Current Snapshot: MvccSnapshot[applied={T|T < 7334205690083942400}]
I20260928 06:39:21.075188 23427 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:39:21.075424 23427 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:39:21.077930 23427 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4320bd0109e54981b118e6ec2c55f933: Sending DeleteTablet for 3 replicas of tablet 22415ebf846546289160d8b1e5d5d57e
I20260928 06:39:21.078846 23824 tablet_service.cc:1514] Processing DeleteTablet for tablet 22415ebf846546289160d8b1e5d5d57e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:21 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:55268
I20260928 06:39:21.078874 23552 tablet_service.cc:1514] Processing DeleteTablet for tablet 22415ebf846546289160d8b1e5d5d57e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:21 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:47588
I20260928 06:39:21.079411 23689 tablet_service.cc:1514] Processing DeleteTablet for tablet 22415ebf846546289160d8b1e5d5d57e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:21 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:47824
I20260928 06:39:21.079947 23933 tablet_replica.cc:330] T 22415ebf846546289160d8b1e5d5d57e P b6a469f7238346c889533948f9298c31: stopping tablet replica
I20260928 06:39:21.080170 23931 tablet_replica.cc:330] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7: stopping tablet replica
I20260928 06:39:21.080363 23933 raft_consensus.cc:2229] T 22415ebf846546289160d8b1e5d5d57e P b6a469f7238346c889533948f9298c31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:39:21.080417 23931 raft_consensus.cc:2229] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:39:21.080703 23933 raft_consensus.cc:2258] T 22415ebf846546289160d8b1e5d5d57e P b6a469f7238346c889533948f9298c31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:21.080801 23931 raft_consensus.cc:2258] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:21.080873 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 23489
I20260928 06:39:21.083590 23931 ts_tablet_manager.cc:1905] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:39:21.085872 23931 ts_tablet_manager.cc:1918] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:39:21.085997 23931 log.cc:1196] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/22415ebf846546289160d8b1e5d5d57e
I20260928 06:39:21.086669 23931 ts_tablet_manager.cc:1939] T 22415ebf846546289160d8b1e5d5d57e P 3e62ef28eb534f2d88b5f4c4aa3af3e7: Deleting consensus metadata
I20260928 06:39:21.087898 23412 catalog_manager.cc:4841] TS 3e62ef28eb534f2d88b5f4c4aa3af3e7 (127.5.22.3:42583): tablet 22415ebf846546289160d8b1e5d5d57e (table test-table [id=efb09215095b494c8afb9d5e9834947d]) successfully deleted
I20260928 06:39:21.087947 23933 ts_tablet_manager.cc:1905] T 22415ebf846546289160d8b1e5d5d57e P b6a469f7238346c889533948f9298c31: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:39:21.089665 23411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.22.1:42579 (error 108)
I20260928 06:39:21.090509 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 23625
I20260928 06:39:21.090536 23933 ts_tablet_manager.cc:1918] T 22415ebf846546289160d8b1e5d5d57e P b6a469f7238346c889533948f9298c31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:39:21.090631 23933 log.cc:1196] T 22415ebf846546289160d8b1e5d5d57e P b6a469f7238346c889533948f9298c31: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577308481727-5208-0/minicluster-data/ts-1/wal/wals/22415ebf846546289160d8b1e5d5d57e
I20260928 06:39:21.090814 23904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:21.090474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:42579 (local address 127.0.0.1:47592)
0928 06:39:21.090623 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:21.090710 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.1:42579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 06:39:21.092324 23411 catalog_manager.cc:4584] TS 60a44cf0ecd24cf4af25477f0fe8dda6 (127.5.22.1:42579): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 22415ebf846546289160d8b1e5d5d57e: Network error: Client connection negotiation failed: client connection to 127.5.22.1:42579: connect: Connection refused (error 111)
I20260928 06:39:21.097707 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 23761
I20260928 06:39:21.099018 23904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:21.098028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:34235 (local address 127.0.0.1:47828)
0928 06:39:21.098826 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:21.098918 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.2:34235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":695}
I20260928 06:39:21.104151 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 23395
2026-09-28T06:39:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2384 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T06:39:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:21Z Disabled control of system clock
I20260928 06:39:21.135473 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:34633
--webserver_interface=127.5.22.62
--webserver_port=0
--builtin_ntp_servers=127.5.22.20:35915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:34633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:39:21.247120 23938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:21.247406 23938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:21.247440 23938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:21.251613 23938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:21.251677 23938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:21.251691 23938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:21.251708 23938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:21.251726 23938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:21.256595 23938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:34633
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:34633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:21.257751 23938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:21.259196 23938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:21.267809 23944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:21.267890 23943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:21.267926 23938 server_base.cc:1034] running on GCE node
W20260928 06:39:21.267796 23946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:21.268779 23938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:21.269968 23938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:21.271180 23938 hybrid_clock.cc:648] HybridClock initialized: now 1790577561271109 us; error 92 us; skew 500 ppm
I20260928 06:39:21.273161 23938 webserver.cc:460] Webserver started at http://127.5.22.62:38493/ using document root <none> and password file <none>
I20260928 06:39:21.273792 23938 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:21.273902 23938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:21.274140 23938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:21.276381 23938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "7c27c34db1bb497091c3fe3e3f29e61b"
format_stamp: "Formatted at 2026-09-28 06:39:21 on dist-test-slave-j0qf"
server_key: "4317c04bbd55df338e8febc4c4e0e48a"
server_key_iv: "2a73acc10943d5e5444d307db1e5076e"
server_key_version: "encryptionkey@0"
I20260928 06:39:21.277004 23938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "7c27c34db1bb497091c3fe3e3f29e61b"
format_stamp: "Formatted at 2026-09-28 06:39:21 on dist-test-slave-j0qf"
server_key: "4317c04bbd55df338e8febc4c4e0e48a"
server_key_iv: "2a73acc10943d5e5444d307db1e5076e"
server_key_version: "encryptionkey@0"
I20260928 06:39:21.280956 23938 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 06:39:21.283620 23952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:21.284942 23938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:39:21.285110 23938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "7c27c34db1bb497091c3fe3e3f29e61b"
format_stamp: "Formatted at 2026-09-28 06:39:21 on dist-test-slave-j0qf"
server_key: "4317c04bbd55df338e8febc4c4e0e48a"
server_key_iv: "2a73acc10943d5e5444d307db1e5076e"
server_key_version: "encryptionkey@0"
I20260928 06:39:21.285249 23938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:21.307651 23938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:21.308465 23938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:21.308698 23938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:21.316275 23938 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:34633
I20260928 06:39:21.316308 24004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:34633 every 8 connection(s)
I20260928 06:39:21.317348 23938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:21.320559 24005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:21.321740 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 23938
I20260928 06:39:21.321879 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:21.322198 5208 external_mini_cluster.cc:1428] Setting key 693dea61977ff519a4a5c1eeeecacea0
I20260928 06:39:21.326995 24005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c27c34db1bb497091c3fe3e3f29e61b: Bootstrap starting.
I20260928 06:39:21.329237 24005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c27c34db1bb497091c3fe3e3f29e61b: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:21.329999 24005 log.cc:824] T 00000000000000000000000000000000 P 7c27c34db1bb497091c3fe3e3f29e61b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:21.332041 24005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c27c34db1bb497091c3fe3e3f29e61b: No bootstrap required, opened a new log
I20260928 06:39:21.332641 24008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:21.323692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45236 (local address 127.5.22.62:34633)
0928 06:39:21.324380 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:21.324396 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:21.324434 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:39:21.325272 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:21.325299 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:21.325763 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:21.326240 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:21.326725 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:21.327834 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:21.328856 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:21.329461 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:21.329652 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:21.330682 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:21.330688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:21.330707 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:21.331122 (+ 415us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:21.331180 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:21.331202 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:21.331329 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:21.331494 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 06:39:21.331763 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":135,"threads_started":1}
I20260928 06:39:21.335964 24005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c27c34db1bb497091c3fe3e3f29e61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c27c34db1bb497091c3fe3e3f29e61b" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34633 } }
I20260928 06:39:21.336196 24005 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c27c34db1bb497091c3fe3e3f29e61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:21.336237 24005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c27c34db1bb497091c3fe3e3f29e61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c27c34db1bb497091c3fe3e3f29e61b, State: Initialized, Role: FOLLOWER
I20260928 06:39:21.336585 24005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c27c34db1bb497091c3fe3e3f29e61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c27c34db1bb497091c3fe3e3f29e61b" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34633 } }
I20260928 06:39:21.336694 24005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c27c34db1bb497091c3fe3e3f29e61b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:21.336731 24005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c27c34db1bb497091c3fe3e3f29e61b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:21.336798 24005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c27c34db1bb497091c3fe3e3f29e61b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:21.337618 24005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c27c34db1bb497091c3fe3e3f29e61b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c27c34db1bb497091c3fe3e3f29e61b" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34633 } }
I20260928 06:39:21.337899 24005 leader_election.cc:302] T 00000000000000000000000000000000 P 7c27c34db1bb497091c3fe3e3f29e61b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c27c34db1bb497091c3fe3e3f29e61b; no voters:
I20260928 06:39:21.338160 24005 leader_election.cc:288] T 00000000000000000000000000000000 P 7c27c34db1bb497091c3fe3e3f29e61b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:21.338340 24010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c27c34db1bb497091c3fe3e3f29e61b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:21.338616 24010 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c27c34db1bb497091c3fe3e3f29e61b [term 1 LEADER]: Becoming Leader. State: Replica: 7c27c34db1bb497091c3fe3e3f29e61b, State: Running, Role: LEADER
I20260928 06:39:21.338945 24010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c27c34db1bb497091c3fe3e3f29e61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c27c34db1bb497091c3fe3e3f29e61b" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34633 } }
I20260928 06:39:21.339383 24005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c27c34db1bb497091c3fe3e3f29e61b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:21.340670 24012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c27c34db1bb497091c3fe3e3f29e61b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c27c34db1bb497091c3fe3e3f29e61b. Latest consensus state: current_term: 1 leader_uuid: "7c27c34db1bb497091c3fe3e3f29e61b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c27c34db1bb497091c3fe3e3f29e61b" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34633 } } }
I20260928 06:39:21.340819 24012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c27c34db1bb497091c3fe3e3f29e61b [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:21.341146 24011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c27c34db1bb497091c3fe3e3f29e61b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c27c34db1bb497091c3fe3e3f29e61b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c27c34db1bb497091c3fe3e3f29e61b" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 34633 } } }
I20260928 06:39:21.341284 24011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c27c34db1bb497091c3fe3e3f29e61b [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:21.341382 24020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:21.344269 24020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:21.350003 24020 catalog_manager.cc:1347] Generated new cluster ID: 854156765aed472198f5c59cf97ee8ce
I20260928 06:39:21.350085 24020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:21.374251 24020 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:21.375223 24020 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:21.385003 24020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c27c34db1bb497091c3fe3e3f29e61b: Generated new TSK 0
I20260928 06:39:21.385723 24020 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:21.395568 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:34633
--builtin_ntp_servers=127.5.22.20:35915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:39:21.508011 24029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:21.508277 24029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:21.508315 24029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:21.512454 24029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:21.512514 24029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:21.512569 24029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:39:21.517369 24029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:34633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:21.518517 24029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:21.519922 24029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:21.529723 24035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:21.529718 24036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:21.529718 24038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:21.529902 24029 server_base.cc:1034] running on GCE node
I20260928 06:39:21.530510 24029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:21.531109 24029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:21.532311 24029 hybrid_clock.cc:648] HybridClock initialized: now 1790577561532280 us; error 40 us; skew 500 ppm
I20260928 06:39:21.534241 24029 webserver.cc:460] Webserver started at http://127.5.22.1:39663/ using document root <none> and password file <none>
I20260928 06:39:21.534868 24029 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:21.534968 24029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:21.535208 24029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:21.537322 24029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "21ff864461f7434d9022d445c9b40e12"
format_stamp: "Formatted at 2026-09-28 06:39:21 on dist-test-slave-j0qf"
server_key: "7398d9efd78eebd931e8c3fbf2750bad"
server_key_iv: "28a9225f6c52daf0581982f776be8d50"
server_key_version: "encryptionkey@0"
I20260928 06:39:21.537925 24029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "21ff864461f7434d9022d445c9b40e12"
format_stamp: "Formatted at 2026-09-28 06:39:21 on dist-test-slave-j0qf"
server_key: "7398d9efd78eebd931e8c3fbf2750bad"
server_key_iv: "28a9225f6c52daf0581982f776be8d50"
server_key_version: "encryptionkey@0"
I20260928 06:39:21.541934 24029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 06:39:21.544418 24044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:21.545646 24029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:39:21.545786 24029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "21ff864461f7434d9022d445c9b40e12"
format_stamp: "Formatted at 2026-09-28 06:39:21 on dist-test-slave-j0qf"
server_key: "7398d9efd78eebd931e8c3fbf2750bad"
server_key_iv: "28a9225f6c52daf0581982f776be8d50"
server_key_version: "encryptionkey@0"
I20260928 06:39:21.545924 24029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:21.580217 24029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:21.581033 24029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:21.581249 24029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:21.581909 24029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:21.582962 24029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:21.583025 24029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:21.583076 24029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:21.583096 24029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:21.592396 24029 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:42119
I20260928 06:39:21.592440 24157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:42119 every 8 connection(s)
I20260928 06:39:21.593384 24029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:39:21.602596 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 24029
I20260928 06:39:21.602742 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:39:21.602898 24008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:21.595246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:38077 (local address 127.5.22.62:34633)
0928 06:39:21.595393 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:21.595397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:21.596499 (+ 1102us) server_negotiation.cc:407] Connection header received
0928 06:39:21.597917 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:21.597921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:21.597986 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:21.598079 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:21.599064 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:21.599617 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:21.600811 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:21.601049 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:21.601168 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:21.602372 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:21.602376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:21.602379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:21.602391 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:21.602419 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:21.602428 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:21.602515 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:21.602677 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 06:39:21.602744 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 06:39:21.603106 5208 external_mini_cluster.cc:1428] Setting key 59b2f3c5fda4c1f31bc2e9d1d85f2187
I20260928 06:39:21.603894 24160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:21.595670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:34633 (local address 127.5.22.1:38077)
0928 06:39:21.596349 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:21.596389 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:21.597695 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:21.598268 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:21.598275 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:21.598427 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:21.598920 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:21.598931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:21.599787 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:21.599791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:21.600671 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:21.600680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:21.601210 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:21.601214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:21.601411 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:21.602067 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:39:21.602069 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:21.602106 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:39:21.602167 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:21.602517 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:21.602530 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:21.602537 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 06:39:21.602675 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 06:39:21.602946 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":129,"threads_started":1}
I20260928 06:39:21.605266 24158 heartbeater.cc:347] Connected to a master server at 127.5.22.62:34633
I20260928 06:39:21.605291 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:34633
--builtin_ntp_servers=127.5.22.20:35915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:39:21.605469 24158 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:21.605963 24158 heartbeater.cc:511] Master 127.5.22.62:34633 requested a full tablet report, sending...
I20260928 06:39:21.607517 23969 ts_manager.cc:194] Registered new tserver with Master: 21ff864461f7434d9022d445c9b40e12 (127.5.22.1:42119)
I20260928 06:39:21.608639 23969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.22.1:38077
W20260928 06:39:21.717214 24161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:21.717476 24161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:21.717517 24161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:21.722060 24161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:21.722136 24161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:21.722230 24161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:39:21.727799 24161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:34633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:21.728825 24161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:21.730175 24161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:39:21.740429 24161 server_base.cc:1034] running on GCE node
W20260928 06:39:21.740446 24169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:21.740417 24166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:21.740446 24167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:21.741138 24161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:21.741993 24161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:21.743216 24161 hybrid_clock.cc:648] HybridClock initialized: now 1790577561743180 us; error 59 us; skew 500 ppm
I20260928 06:39:21.745254 24161 webserver.cc:460] Webserver started at http://127.5.22.2:39843/ using document root <none> and password file <none>
I20260928 06:39:21.745869 24161 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:21.745970 24161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:21.746208 24161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:21.748296 24161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "5274aa49d19c41bdb1cddc1ad4cccc2c"
format_stamp: "Formatted at 2026-09-28 06:39:21 on dist-test-slave-j0qf"
server_key: "8d490a5dad1a550968db95d55cf6f92b"
server_key_iv: "2aa121de82126eda8d4d82d764161437"
server_key_version: "encryptionkey@0"
I20260928 06:39:21.748970 24161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "5274aa49d19c41bdb1cddc1ad4cccc2c"
format_stamp: "Formatted at 2026-09-28 06:39:21 on dist-test-slave-j0qf"
server_key: "8d490a5dad1a550968db95d55cf6f92b"
server_key_iv: "2aa121de82126eda8d4d82d764161437"
server_key_version: "encryptionkey@0"
I20260928 06:39:21.753186 24161 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 06:39:21.755657 24176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:21.756902 24161 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:39:21.757041 24161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "5274aa49d19c41bdb1cddc1ad4cccc2c"
format_stamp: "Formatted at 2026-09-28 06:39:21 on dist-test-slave-j0qf"
server_key: "8d490a5dad1a550968db95d55cf6f92b"
server_key_iv: "2aa121de82126eda8d4d82d764161437"
server_key_version: "encryptionkey@0"
I20260928 06:39:21.757159 24161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:21.773849 24161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:21.774657 24161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:21.774871 24161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:21.775547 24161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:21.776544 24161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:21.776623 24161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:21.776690 24161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:21.776738 24161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:21.786063 24161 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:45527
I20260928 06:39:21.786085 24289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:45527 every 8 connection(s)
I20260928 06:39:21.787153 24161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:39:21.792330 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 24161
I20260928 06:39:21.792464 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:39:21.792760 5208 external_mini_cluster.cc:1428] Setting key a763207787307f2342f1bfff76dcd301
I20260928 06:39:21.794912 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:34633
--builtin_ntp_servers=127.5.22.20:35915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:39:21.797875 24008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:21.789060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:49469 (local address 127.5.22.62:34633)
0928 06:39:21.789214 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:21.789219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:21.790465 (+ 1246us) server_negotiation.cc:407] Connection header received
0928 06:39:21.791946 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:21.791949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:21.791994 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:21.792065 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:21.793345 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:21.794075 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:21.795796 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:21.796017 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:21.796095 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:21.797367 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:21.797371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:21.797374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:21.797386 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:21.797415 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:21.797424 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:21.797507 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:21.797676 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 06:39:21.797740 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 06:39:21.798859 24292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:21.789502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:34633 (local address 127.5.22.2:49469)
0928 06:39:21.790259 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:21.790302 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:21.791741 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:21.792310 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:21.792320 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:21.792440 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:21.793166 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:21.793187 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:21.794209 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:21.794214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:21.795658 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:21.795673 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:21.796173 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:21.796179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:21.796414 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:21.797108 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:39:21.797109 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:21.797149 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:39:21.797210 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:21.797522 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:21.797531 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:21.797537 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:39:21.797675 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 06:39:21.797973 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":112,"threads_started":1}
I20260928 06:39:21.800165 24290 heartbeater.cc:347] Connected to a master server at 127.5.22.62:34633
I20260928 06:39:21.800465 24290 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:21.801013 24290 heartbeater.cc:511] Master 127.5.22.62:34633 requested a full tablet report, sending...
I20260928 06:39:21.802189 23969 ts_manager.cc:194] Registered new tserver with Master: 5274aa49d19c41bdb1cddc1ad4cccc2c (127.5.22.2:45527)
I20260928 06:39:21.802809 23969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.22.2:49469
W20260928 06:39:21.908072 24293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:21.908394 24293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:21.908473 24293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:21.912601 24293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:21.912695 24293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:21.912786 24293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:39:21.918087 24293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:34633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:21.919343 24293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:21.920748 24293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:21.930547 24298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:21.930603 24301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:21.930619 24293 server_base.cc:1034] running on GCE node
W20260928 06:39:21.930553 24299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:21.931201 24293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:21.931934 24293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:21.933144 24293 hybrid_clock.cc:648] HybridClock initialized: now 1790577561933131 us; error 38 us; skew 500 ppm
I20260928 06:39:21.935178 24293 webserver.cc:460] Webserver started at http://127.5.22.3:46869/ using document root <none> and password file <none>
I20260928 06:39:21.935791 24293 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:21.935925 24293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:21.936168 24293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:21.938249 24293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "7e28937fb36b4621bf5694aedd06a856"
format_stamp: "Formatted at 2026-09-28 06:39:21 on dist-test-slave-j0qf"
server_key: "e03d8ae54cb2d267d9b9e855fc523683"
server_key_iv: "31de9f0f875989b20f00c178dbe2bb79"
server_key_version: "encryptionkey@0"
I20260928 06:39:21.938910 24293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "7e28937fb36b4621bf5694aedd06a856"
format_stamp: "Formatted at 2026-09-28 06:39:21 on dist-test-slave-j0qf"
server_key: "e03d8ae54cb2d267d9b9e855fc523683"
server_key_iv: "31de9f0f875989b20f00c178dbe2bb79"
server_key_version: "encryptionkey@0"
I20260928 06:39:21.942953 24293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 06:39:21.945392 24307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:21.946732 24293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:39:21.946884 24293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "7e28937fb36b4621bf5694aedd06a856"
format_stamp: "Formatted at 2026-09-28 06:39:21 on dist-test-slave-j0qf"
server_key: "e03d8ae54cb2d267d9b9e855fc523683"
server_key_iv: "31de9f0f875989b20f00c178dbe2bb79"
server_key_version: "encryptionkey@0"
I20260928 06:39:21.947001 24293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:21.969991 24293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:21.970769 24293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:21.970993 24293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:21.971668 24293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:21.972893 24293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:21.973001 24293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:21.973078 24293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:21.973140 24293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:21.983294 24293 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:39809
I20260928 06:39:21.983309 24422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:39809 every 8 connection(s)
I20260928 06:39:21.984342 24293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:39:21.992120 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 24293
I20260928 06:39:21.992340 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
I20260928 06:39:21.992643 5208 external_mini_cluster.cc:1428] Setting key ca17a0cf6698f84df393c27fd6781ca9
I20260928 06:39:21.995549 24008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:21.986496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:55379 (local address 127.5.22.62:34633)
0928 06:39:21.986648 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:21.986651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:21.987930 (+ 1279us) server_negotiation.cc:407] Connection header received
0928 06:39:21.989553 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:21.989557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:21.989621 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:21.989725 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:21.990843 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:21.991392 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:21.993223 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:21.993465 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:21.993640 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:21.995044 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:21.995049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:21.995057 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:21.995070 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:21.995090 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:21.995099 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:21.995176 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:21.995382 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 06:39:21.995439 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 06:39:21.996429 24425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:21.986991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:34633 (local address 127.5.22.3:55379)
0928 06:39:21.987725 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:21.987769 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:21.989307 (+ 1538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:21.990008 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:21.990015 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:21.990107 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:21.990679 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:21.990693 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:21.991539 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:21.991542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:21.993049 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:21.993059 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:21.993701 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:21.993706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:21.993917 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:21.994677 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:39:21.994679 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:21.994753 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:39:21.994856 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:21.995195 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:21.995203 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:21.995210 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 06:39:21.995311 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 06:39:21.995547 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":124,"threads_started":1}
I20260928 06:39:21.997691 24423 heartbeater.cc:347] Connected to a master server at 127.5.22.62:34633
I20260928 06:39:21.997996 24423 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:21.998723 24423 heartbeater.cc:511] Master 127.5.22.62:34633 requested a full tablet report, sending...
I20260928 06:39:21.999904 23969 ts_manager.cc:194] Registered new tserver with Master: 7e28937fb36b4621bf5694aedd06a856 (127.5.22.3:39809)
I20260928 06:39:22.000502 23969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.22.3:55379
I20260928 06:39:22.007141 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:39:22.013630 24008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:22.009362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45248 (local address 127.5.22.62:34633)
0928 06:39:22.009536 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:22.009541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:22.009682 (+ 141us) server_negotiation.cc:407] Connection header received
0928 06:39:22.009869 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:22.009871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:22.009913 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:22.010015 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:22.010301 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:22.010881 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:22.011785 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:22.012019 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:22.012156 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:22.013103 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:22.013107 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:22.013109 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:22.013127 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:22.013146 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:22.013154 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:22.013293 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:22.013430 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 06:39:22.013471 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 06:39:22.017433 23969 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:39:22.020792 23969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:39:22.036629 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:22.030162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:45527 (local address 127.0.0.1:56844)
0928 06:39:22.030888 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:22.030966 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:22.031111 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:22.032325 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:22.032328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:22.032354 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:22.032458 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:22.032464 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:22.033881 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:22.033884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:22.035200 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:22.035212 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:22.036292 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:22.036298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:22.036389 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:22.036408 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 06:39:22.036465 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 06:39:22.036517 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":124,"spinlock_wait_cycles":5376,"thread_start_us":95,"threads_started":1}
I20260928 06:39:22.036697 24436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:22.030315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34016 (local address 127.5.22.3:39809)
0928 06:39:22.031704 (+ 1389us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:22.031715 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:22.031734 (+ 19us) server_negotiation.cc:407] Connection header received
0928 06:39:22.031795 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:22.031799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:22.031952 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:22.032475 (+ 523us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:22.032628 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:22.033353 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:22.035003 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:22.035864 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:22.036134 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:22.036357 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:22.036411 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 06:39:22.036519 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1264,"thread_start_us":157,"threads_started":1}
I20260928 06:39:22.036629 24435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:22.030169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:39809 (local address 127.0.0.1:34016)
0928 06:39:22.030925 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:22.030967 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:22.031110 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:22.032118 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:22.032124 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:22.032178 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:22.032319 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:22.032326 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:22.033476 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:22.033482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:22.034879 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:22.034892 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:22.036018 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:22.036022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:22.036163 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:22.036206 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 06:39:22.036311 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 06:39:22.036438 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":18,"spinlock_wait_cycles":12800,"thread_start_us":167,"threads_started":1}
I20260928 06:39:22.037307 24438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:22.030961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:42119 (local address 127.0.0.1:50598)
0928 06:39:22.031351 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:22.031361 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:22.031467 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:22.032120 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:22.032126 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:22.032187 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:22.032325 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:22.032334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:22.033869 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:22.033872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:22.035112 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:22.035122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:22.035978 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:22.035982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:22.036689 (+ 707us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:22.036886 (+ 197us) client_negotiation.cc:769] Sending connection context
0928 06:39:22.036965 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:39:22.037165 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":100,"spinlock_wait_cycles":10240,"thread_start_us":93,"threads_started":1}
I20260928 06:39:22.037312 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:22.031186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50598 (local address 127.5.22.1:42119)
0928 06:39:22.031592 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:22.031605 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:22.031631 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:39:22.031709 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:22.031716 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:22.031927 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:22.032129 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:22.032463 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:22.033691 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:22.035251 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:22.035834 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:22.035980 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:22.036199 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:22.037030 (+ 831us) server_negotiation.cc:299] Negotiation successful
0928 06:39:22.037143 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":161,"threads_started":1}
I20260928 06:39:22.036738 24437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:22.030535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56844 (local address 127.5.22.2:45527)
0928 06:39:22.031796 (+ 1261us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:22.031811 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:22.031838 (+ 27us) server_negotiation.cc:407] Connection header received
0928 06:39:22.031921 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:22.031928 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:22.032126 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:22.032412 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:22.032564 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:22.033701 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:22.035352 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:22.036130 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:22.036285 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:22.036456 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:22.036500 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 06:39:22.036575 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1152,"thread_start_us":118,"threads_started":1}
I20260928 06:39:22.040514 24092 tablet_service.cc:1467] Processing CreateTablet for tablet 4adcf05f7ac648a88d42f9608379d42f (DEFAULT_TABLE table=test-table [id=32bc69695cb84b2f86e7696e69f883bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:22.040514 24356 tablet_service.cc:1467] Processing CreateTablet for tablet 4adcf05f7ac648a88d42f9608379d42f (DEFAULT_TABLE table=test-table [id=32bc69695cb84b2f86e7696e69f883bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:22.040742 24224 tablet_service.cc:1467] Processing CreateTablet for tablet 4adcf05f7ac648a88d42f9608379d42f (DEFAULT_TABLE table=test-table [id=32bc69695cb84b2f86e7696e69f883bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:22.041666 24356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4adcf05f7ac648a88d42f9608379d42f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:22.041666 24092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4adcf05f7ac648a88d42f9608379d42f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:22.041793 24224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4adcf05f7ac648a88d42f9608379d42f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:22.048300 24440 tablet_bootstrap.cc:492] T 4adcf05f7ac648a88d42f9608379d42f P 5274aa49d19c41bdb1cddc1ad4cccc2c: Bootstrap starting.
I20260928 06:39:22.049371 24441 tablet_bootstrap.cc:492] T 4adcf05f7ac648a88d42f9608379d42f P 21ff864461f7434d9022d445c9b40e12: Bootstrap starting.
I20260928 06:39:22.050642 24442 tablet_bootstrap.cc:492] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856: Bootstrap starting.
I20260928 06:39:22.050912 24440 tablet_bootstrap.cc:654] T 4adcf05f7ac648a88d42f9608379d42f P 5274aa49d19c41bdb1cddc1ad4cccc2c: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:22.051558 24441 tablet_bootstrap.cc:654] T 4adcf05f7ac648a88d42f9608379d42f P 21ff864461f7434d9022d445c9b40e12: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:22.051782 24440 log.cc:824] T 4adcf05f7ac648a88d42f9608379d42f P 5274aa49d19c41bdb1cddc1ad4cccc2c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:22.052372 24441 log.cc:824] T 4adcf05f7ac648a88d42f9608379d42f P 21ff864461f7434d9022d445c9b40e12: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:22.052747 24442 tablet_bootstrap.cc:654] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:22.053439 24442 log.cc:824] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:22.054421 24441 tablet_bootstrap.cc:492] T 4adcf05f7ac648a88d42f9608379d42f P 21ff864461f7434d9022d445c9b40e12: No bootstrap required, opened a new log
I20260928 06:39:22.054421 24440 tablet_bootstrap.cc:492] T 4adcf05f7ac648a88d42f9608379d42f P 5274aa49d19c41bdb1cddc1ad4cccc2c: No bootstrap required, opened a new log
I20260928 06:39:22.054616 24441 ts_tablet_manager.cc:1397] T 4adcf05f7ac648a88d42f9608379d42f P 21ff864461f7434d9022d445c9b40e12: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 06:39:22.054616 24440 ts_tablet_manager.cc:1397] T 4adcf05f7ac648a88d42f9608379d42f P 5274aa49d19c41bdb1cddc1ad4cccc2c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 06:39:22.055202 24442 tablet_bootstrap.cc:492] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856: No bootstrap required, opened a new log
I20260928 06:39:22.055406 24442 ts_tablet_manager.cc:1397] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 06:39:22.058516 24442 raft_consensus.cc:348] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e28937fb36b4621bf5694aedd06a856" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 39809 } } peers { permanent_uuid: "5274aa49d19c41bdb1cddc1ad4cccc2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 45527 } } peers { permanent_uuid: "21ff864461f7434d9022d445c9b40e12" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42119 } }
I20260928 06:39:22.058760 24442 raft_consensus.cc:374] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:22.058812 24442 raft_consensus.cc:729] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e28937fb36b4621bf5694aedd06a856, State: Initialized, Role: FOLLOWER
I20260928 06:39:22.058933 24440 raft_consensus.cc:348] T 4adcf05f7ac648a88d42f9608379d42f P 5274aa49d19c41bdb1cddc1ad4cccc2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e28937fb36b4621bf5694aedd06a856" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 39809 } } peers { permanent_uuid: "5274aa49d19c41bdb1cddc1ad4cccc2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 45527 } } peers { permanent_uuid: "21ff864461f7434d9022d445c9b40e12" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42119 } }
I20260928 06:39:22.058933 24441 raft_consensus.cc:348] T 4adcf05f7ac648a88d42f9608379d42f P 21ff864461f7434d9022d445c9b40e12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e28937fb36b4621bf5694aedd06a856" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 39809 } } peers { permanent_uuid: "5274aa49d19c41bdb1cddc1ad4cccc2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 45527 } } peers { permanent_uuid: "21ff864461f7434d9022d445c9b40e12" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42119 } }
I20260928 06:39:22.059182 24441 raft_consensus.cc:374] T 4adcf05f7ac648a88d42f9608379d42f P 21ff864461f7434d9022d445c9b40e12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:22.059182 24440 raft_consensus.cc:374] T 4adcf05f7ac648a88d42f9608379d42f P 5274aa49d19c41bdb1cddc1ad4cccc2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:22.059232 24440 raft_consensus.cc:729] T 4adcf05f7ac648a88d42f9608379d42f P 5274aa49d19c41bdb1cddc1ad4cccc2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5274aa49d19c41bdb1cddc1ad4cccc2c, State: Initialized, Role: FOLLOWER
I20260928 06:39:22.059232 24441 raft_consensus.cc:729] T 4adcf05f7ac648a88d42f9608379d42f P 21ff864461f7434d9022d445c9b40e12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21ff864461f7434d9022d445c9b40e12, State: Initialized, Role: FOLLOWER
I20260928 06:39:22.059284 24442 consensus_queue.cc:260] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e28937fb36b4621bf5694aedd06a856" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 39809 } } peers { permanent_uuid: "5274aa49d19c41bdb1cddc1ad4cccc2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 45527 } } peers { permanent_uuid: "21ff864461f7434d9022d445c9b40e12" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42119 } }
I20260928 06:39:22.059660 24440 consensus_queue.cc:260] T 4adcf05f7ac648a88d42f9608379d42f P 5274aa49d19c41bdb1cddc1ad4cccc2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e28937fb36b4621bf5694aedd06a856" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 39809 } } peers { permanent_uuid: "5274aa49d19c41bdb1cddc1ad4cccc2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 45527 } } peers { permanent_uuid: "21ff864461f7434d9022d445c9b40e12" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42119 } }
I20260928 06:39:22.059660 24441 consensus_queue.cc:260] T 4adcf05f7ac648a88d42f9608379d42f P 21ff864461f7434d9022d445c9b40e12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e28937fb36b4621bf5694aedd06a856" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 39809 } } peers { permanent_uuid: "5274aa49d19c41bdb1cddc1ad4cccc2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 45527 } } peers { permanent_uuid: "21ff864461f7434d9022d445c9b40e12" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42119 } }
I20260928 06:39:22.060498 24442 ts_tablet_manager.cc:1428] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 06:39:22.060634 24441 ts_tablet_manager.cc:1428] T 4adcf05f7ac648a88d42f9608379d42f P 21ff864461f7434d9022d445c9b40e12: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 06:39:22.060720 24440 ts_tablet_manager.cc:1428] T 4adcf05f7ac648a88d42f9608379d42f P 5274aa49d19c41bdb1cddc1ad4cccc2c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 06:39:22.060760 24423 heartbeater.cc:503] Master 127.5.22.62:34633 was elected leader, sending a full tablet report...
I20260928 06:39:22.060981 24158 heartbeater.cc:503] Master 127.5.22.62:34633 was elected leader, sending a full tablet report...
I20260928 06:39:22.061977 24290 heartbeater.cc:503] Master 127.5.22.62:34633 was elected leader, sending a full tablet report...
W20260928 06:39:22.094558 24159 tablet.cc:2369] T 4adcf05f7ac648a88d42f9608379d42f P 21ff864461f7434d9022d445c9b40e12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:39:22.137598 24447 raft_consensus.cc:482] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:39:22.137874 24447 raft_consensus.cc:504] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e28937fb36b4621bf5694aedd06a856" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 39809 } } peers { permanent_uuid: "5274aa49d19c41bdb1cddc1ad4cccc2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 45527 } } peers { permanent_uuid: "21ff864461f7434d9022d445c9b40e12" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42119 } }
I20260928 06:39:22.139227 24447 leader_election.cc:288] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5274aa49d19c41bdb1cddc1ad4cccc2c (127.5.22.2:45527), 21ff864461f7434d9022d445c9b40e12 (127.5.22.1:42119)
I20260928 06:39:22.144567 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:22.140044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:47515 (local address 127.5.22.1:42119)
0928 06:39:22.140208 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:22.140214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:22.140440 (+ 226us) server_negotiation.cc:407] Connection header received
0928 06:39:22.140575 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:22.140579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:22.140632 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:22.140735 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:22.140992 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:22.141677 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:22.143141 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:22.143807 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:22.143932 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:22.143994 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:22.144353 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 06:39:22.144432 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 06:39:22.144567 24437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:22.139674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:36391 (local address 127.5.22.2:45527)
0928 06:39:22.139828 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:22.139833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:22.139849 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:39:22.140074 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:22.140080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:22.140143 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:22.140226 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:22.140599 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:22.141444 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:22.143239 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:22.143819 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:22.143932 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:22.143994 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:22.144353 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 06:39:22.144432 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 06:39:22.144611 24425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:22.139588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:45527 (local address 127.5.22.3:36391)
0928 06:39:22.139758 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:22.139776 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:22.139929 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:22.140284 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:22.140287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:22.140305 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:22.140454 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:22.140462 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:22.141556 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:22.141558 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:22.142973 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:22.142984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:22.143954 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:22.143958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:22.144222 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:22.144239 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 06:39:22.144305 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:39:22.144449 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":146}
I20260928 06:39:22.144619 24450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:22.139900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:42119 (local address 127.5.22.3:47515)
0928 06:39:22.140316 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:22.140329 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:22.140443 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:22.140755 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:22.140757 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:22.140772 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:22.140879 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:22.140883 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:22.141770 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:22.141773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:22.142995 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:22.143002 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:22.143955 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:22.143958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:22.144124 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:22.144222 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 06:39:22.144305 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 06:39:22.144449 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":178,"thread_start_us":182,"threads_started":1}
I20260928 06:39:22.145419 24112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4adcf05f7ac648a88d42f9608379d42f" candidate_uuid: "7e28937fb36b4621bf5694aedd06a856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ff864461f7434d9022d445c9b40e12" is_pre_election: true
I20260928 06:39:22.145419 24244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4adcf05f7ac648a88d42f9608379d42f" candidate_uuid: "7e28937fb36b4621bf5694aedd06a856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5274aa49d19c41bdb1cddc1ad4cccc2c" is_pre_election: true
I20260928 06:39:22.145762 24112 raft_consensus.cc:2454] T 4adcf05f7ac648a88d42f9608379d42f P 21ff864461f7434d9022d445c9b40e12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e28937fb36b4621bf5694aedd06a856 in term 0.
I20260928 06:39:22.145762 24244 raft_consensus.cc:2454] T 4adcf05f7ac648a88d42f9608379d42f P 5274aa49d19c41bdb1cddc1ad4cccc2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e28937fb36b4621bf5694aedd06a856 in term 0.
I20260928 06:39:22.146292 24310 leader_election.cc:302] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21ff864461f7434d9022d445c9b40e12, 7e28937fb36b4621bf5694aedd06a856; no voters:
I20260928 06:39:22.146625 24447 raft_consensus.cc:2781] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:39:22.146739 24447 raft_consensus.cc:482] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:39:22.146803 24447 raft_consensus.cc:3037] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:22.147889 24447 raft_consensus.cc:504] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e28937fb36b4621bf5694aedd06a856" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 39809 } } peers { permanent_uuid: "5274aa49d19c41bdb1cddc1ad4cccc2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 45527 } } peers { permanent_uuid: "21ff864461f7434d9022d445c9b40e12" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42119 } }
I20260928 06:39:22.148248 24447 leader_election.cc:288] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856 [CANDIDATE]: Term 1 election: Requested vote from peers 5274aa49d19c41bdb1cddc1ad4cccc2c (127.5.22.2:45527), 21ff864461f7434d9022d445c9b40e12 (127.5.22.1:42119)
I20260928 06:39:22.148689 24112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4adcf05f7ac648a88d42f9608379d42f" candidate_uuid: "7e28937fb36b4621bf5694aedd06a856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ff864461f7434d9022d445c9b40e12"
I20260928 06:39:22.148743 24244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4adcf05f7ac648a88d42f9608379d42f" candidate_uuid: "7e28937fb36b4621bf5694aedd06a856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5274aa49d19c41bdb1cddc1ad4cccc2c"
I20260928 06:39:22.148854 24112 raft_consensus.cc:3037] T 4adcf05f7ac648a88d42f9608379d42f P 21ff864461f7434d9022d445c9b40e12 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:22.148900 24244 raft_consensus.cc:3037] T 4adcf05f7ac648a88d42f9608379d42f P 5274aa49d19c41bdb1cddc1ad4cccc2c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:22.150056 24112 raft_consensus.cc:2454] T 4adcf05f7ac648a88d42f9608379d42f P 21ff864461f7434d9022d445c9b40e12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e28937fb36b4621bf5694aedd06a856 in term 1.
I20260928 06:39:22.150049 24244 raft_consensus.cc:2454] T 4adcf05f7ac648a88d42f9608379d42f P 5274aa49d19c41bdb1cddc1ad4cccc2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e28937fb36b4621bf5694aedd06a856 in term 1.
I20260928 06:39:22.150415 24310 leader_election.cc:302] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21ff864461f7434d9022d445c9b40e12, 7e28937fb36b4621bf5694aedd06a856; no voters:
I20260928 06:39:22.150698 24447 raft_consensus.cc:2781] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:22.150933 24447 raft_consensus.cc:686] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856 [term 1 LEADER]: Becoming Leader. State: Replica: 7e28937fb36b4621bf5694aedd06a856, State: Running, Role: LEADER
I20260928 06:39:22.151582 24447 consensus_queue.cc:237] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e28937fb36b4621bf5694aedd06a856" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 39809 } } peers { permanent_uuid: "5274aa49d19c41bdb1cddc1ad4cccc2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 45527 } } peers { permanent_uuid: "21ff864461f7434d9022d445c9b40e12" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42119 } }
I20260928 06:39:22.154786 23967 catalog_manager.cc:5495] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e28937fb36b4621bf5694aedd06a856 (127.5.22.3). New cstate: current_term: 1 leader_uuid: "7e28937fb36b4621bf5694aedd06a856" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e28937fb36b4621bf5694aedd06a856" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 39809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5274aa49d19c41bdb1cddc1ad4cccc2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 45527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21ff864461f7434d9022d445c9b40e12" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42119 } health_report { overall_health: UNKNOWN } } }
I20260928 06:39:22.190501 24436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:22.186628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34022 (local address 127.5.22.3:39809)
0928 06:39:22.186775 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:22.186779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:22.186794 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:39:22.186878 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:22.186880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:22.186933 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:22.187022 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:22.187353 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:22.187857 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:22.188694 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:22.188996 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:22.189078 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:22.189520 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:22.189922 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:22.190099 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:22.190194 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 06:39:22.190227 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 06:39:22.196359 24112 raft_consensus.cc:1261] T 4adcf05f7ac648a88d42f9608379d42f P 21ff864461f7434d9022d445c9b40e12 [term 1 FOLLOWER]: Refusing update from remote peer 7e28937fb36b4621bf5694aedd06a856: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:22.196358 24244 raft_consensus.cc:1261] T 4adcf05f7ac648a88d42f9608379d42f P 5274aa49d19c41bdb1cddc1ad4cccc2c [term 1 FOLLOWER]: Refusing update from remote peer 7e28937fb36b4621bf5694aedd06a856: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:22.197122 24447 consensus_queue.cc:1035] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21ff864461f7434d9022d445c9b40e12" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:22.197392 24451 consensus_queue.cc:1035] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5274aa49d19c41bdb1cddc1ad4cccc2c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 45527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:22.206877 24455 mvcc.cc:205] Tried to move back new op lower bound from 7334205694749192192 to 7334205694576152576. Current Snapshot: MvccSnapshot[applied={T|T < 7334205694749192192}]
I20260928 06:39:22.208552 24454 mvcc.cc:205] Tried to move back new op lower bound from 7334205694749192192 to 7334205694576152576. Current Snapshot: MvccSnapshot[applied={T|T < 7334205694749192192 or (T in {7334205694749192192})}]
I20260928 06:39:22.214702 23967 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:39:22.214844 23967 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:39:22.217306 23967 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c27c34db1bb497091c3fe3e3f29e61b: Sending DeleteTablet for 3 replicas of tablet 4adcf05f7ac648a88d42f9608379d42f
I20260928 06:39:22.218106 24356 tablet_service.cc:1514] Processing DeleteTablet for tablet 4adcf05f7ac648a88d42f9608379d42f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:22 UTC) from {username='slave'} at 127.0.0.1:34016
I20260928 06:39:22.218139 24224 tablet_service.cc:1514] Processing DeleteTablet for tablet 4adcf05f7ac648a88d42f9608379d42f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:22 UTC) from {username='slave'} at 127.0.0.1:56844
I20260928 06:39:22.218236 24092 tablet_service.cc:1514] Processing DeleteTablet for tablet 4adcf05f7ac648a88d42f9608379d42f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:22 UTC) from {username='slave'} at 127.0.0.1:50598
I20260928 06:39:22.218796 24465 tablet_replica.cc:330] T 4adcf05f7ac648a88d42f9608379d42f P 21ff864461f7434d9022d445c9b40e12: stopping tablet replica
I20260928 06:39:22.219187 24465 raft_consensus.cc:2229] T 4adcf05f7ac648a88d42f9608379d42f P 21ff864461f7434d9022d445c9b40e12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:39:22.219411 24463 tablet_replica.cc:330] T 4adcf05f7ac648a88d42f9608379d42f P 5274aa49d19c41bdb1cddc1ad4cccc2c: stopping tablet replica
I20260928 06:39:22.219543 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 24029
I20260928 06:39:22.219774 24463 raft_consensus.cc:2229] T 4adcf05f7ac648a88d42f9608379d42f P 5274aa49d19c41bdb1cddc1ad4cccc2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:39:22.220182 24464 tablet_replica.cc:330] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856: stopping tablet replica
I20260928 06:39:22.221535 24464 raft_consensus.cc:2229] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:39:22.221907 24464 raft_consensus.cc:2258] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:22.222198 24463 raft_consensus.cc:2258] T 4adcf05f7ac648a88d42f9608379d42f P 5274aa49d19c41bdb1cddc1ad4cccc2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:22.223578 24464 ts_tablet_manager.cc:1905] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:39:22.225780 24464 ts_tablet_manager.cc:1918] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:39:22.225911 24464 log.cc:1196] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/4adcf05f7ac648a88d42f9608379d42f
I20260928 06:39:22.226199 24464 ts_tablet_manager.cc:1939] T 4adcf05f7ac648a88d42f9608379d42f P 7e28937fb36b4621bf5694aedd06a856: Deleting consensus metadata
I20260928 06:39:22.227569 23954 catalog_manager.cc:4841] TS 7e28937fb36b4621bf5694aedd06a856 (127.5.22.3:39809): tablet 4adcf05f7ac648a88d42f9608379d42f (table test-table [id=32bc69695cb84b2f86e7696e69f883bc]) successfully deleted
W20260928 06:39:22.228219 23955 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.22.1:42119 (error 108)
I20260928 06:39:22.228560 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 24161
I20260928 06:39:22.231017 24438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:22.230727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:42119 (local address 127.0.0.1:50602)
0928 06:39:22.230869 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:22.230943 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.1:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 06:39:22.231655 23955 catalog_manager.cc:4584] TS 21ff864461f7434d9022d445c9b40e12 (127.5.22.1:42119): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4adcf05f7ac648a88d42f9608379d42f: Network error: Client connection negotiation failed: client connection to 127.5.22.1:42119: connect: Connection refused (error 111)
I20260928 06:39:22.237200 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 24293
I20260928 06:39:22.237541 24438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:22.237196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:45527 (local address 127.0.0.1:56852)
0928 06:39:22.237374 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:22.237440 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.2:45527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 06:39:22.243717 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 23938
2026-09-28T06:39:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1138 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:22 dist-test-slave-j0qf krb5kdc[24469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:22 dist-test-slave-j0qf krb5kdc[24469](info): set up 2 sockets
Sep 28 06:39:22 dist-test-slave-j0qf krb5kdc[24469](info): commencing operation
krb5kdc: starting...
W20260928 06:39:23.297827 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:39:23 dist-test-slave-j0qf krb5kdc[24469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:39:23.426009 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:36935
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:36935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:23.535264 24482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:23.535620 24482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:23.535702 24482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:23.539826 24482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:23.539918 24482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:23.539956 24482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:23.540024 24482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:23.540057 24482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:23.545445 24482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:36935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:36935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:23.546672 24482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:23.547711 24482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:23.556514 24488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:23.556720 24487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:23.556739 24490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:23.557042 24482 server_base.cc:1034] running on GCE node
I20260928 06:39:23.557600 24482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:23.558854 24482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:23.560055 24482 hybrid_clock.cc:648] HybridClock initialized: now 1790577563560037 us; error 31 us; skew 500 ppm
Sep 28 06:39:23 dist-test-slave-j0qf krb5kdc[24469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577563, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:23.563014 24482 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:39:23.563984 24482 webserver.cc:460] Webserver started at http://127.5.22.62:46395/ using document root <none> and password file <none>
I20260928 06:39:23.564505 24482 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:23.564563 24482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:23.564726 24482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:23.566845 24482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "59c8a1425b5441988caa98ddee9eafc4"
format_stamp: "Formatted at 2026-09-28 06:39:23 on dist-test-slave-j0qf"
server_key: "c78a2760e42e9ef4f9c81c924b52d4fa"
server_key_iv: "a666144feef8e781662765f1f2671bf0"
server_key_version: "encryptionkey@0"
I20260928 06:39:23.567382 24482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "59c8a1425b5441988caa98ddee9eafc4"
format_stamp: "Formatted at 2026-09-28 06:39:23 on dist-test-slave-j0qf"
server_key: "c78a2760e42e9ef4f9c81c924b52d4fa"
server_key_iv: "a666144feef8e781662765f1f2671bf0"
server_key_version: "encryptionkey@0"
I20260928 06:39:23.571408 24482 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 06:39:23.573804 24497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:23.575281 24482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:39:23.575423 24482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "59c8a1425b5441988caa98ddee9eafc4"
format_stamp: "Formatted at 2026-09-28 06:39:23 on dist-test-slave-j0qf"
server_key: "c78a2760e42e9ef4f9c81c924b52d4fa"
server_key_iv: "a666144feef8e781662765f1f2671bf0"
server_key_version: "encryptionkey@0"
I20260928 06:39:23.575513 24482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:23.596447 24482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:23.599535 24482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:23.599689 24482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:23.607189 24482 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:36935
I20260928 06:39:23.607268 24550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:36935 every 8 connection(s)
I20260928 06:39:23.608197 24482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:23.611222 24551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:23.612891 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 24482
I20260928 06:39:23.613044 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:23.613397 5208 external_mini_cluster.cc:1428] Setting key eda00d4ace04b4ded3e236b86178fed0
I20260928 06:39:23.618037 24551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59c8a1425b5441988caa98ddee9eafc4: Bootstrap starting.
I20260928 06:39:23.620476 24551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59c8a1425b5441988caa98ddee9eafc4: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:23.621340 24551 log.cc:824] T 00000000000000000000000000000000 P 59c8a1425b5441988caa98ddee9eafc4: Log is configured to *not* fsync() on all Append() calls
Sep 28 06:39:23 dist-test-slave-j0qf krb5kdc[24469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:39:23.623979 24551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59c8a1425b5441988caa98ddee9eafc4: No bootstrap required, opened a new log
I20260928 06:39:23.627971 24551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59c8a1425b5441988caa98ddee9eafc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c8a1425b5441988caa98ddee9eafc4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 36935 } }
I20260928 06:39:23.628279 24551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59c8a1425b5441988caa98ddee9eafc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:23.628361 24551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59c8a1425b5441988caa98ddee9eafc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59c8a1425b5441988caa98ddee9eafc4, State: Initialized, Role: FOLLOWER
I20260928 06:39:23.628829 24551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59c8a1425b5441988caa98ddee9eafc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c8a1425b5441988caa98ddee9eafc4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 36935 } }
I20260928 06:39:23.628952 24551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59c8a1425b5441988caa98ddee9eafc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:23.629040 24551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59c8a1425b5441988caa98ddee9eafc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:23.629161 24551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59c8a1425b5441988caa98ddee9eafc4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:23.629292 24554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:23.615115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41354 (local address 127.5.22.62:36935)
0928 06:39:23.615642 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:23.615657 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:23.615692 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:39:23.616371 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:23.616389 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:23.616714 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:23.617067 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:23.618083 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:23.619256 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:23.620136 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:23.620704 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:23.620887 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:23.624037 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:23.624042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:23.624084 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:23.624119 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:23.626106 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:23.626643 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:23.626649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:23.626653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:23.626779 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:23.627089 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:23.627092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:23.627103 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:23.627492 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:23.627722 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:23.628014 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 06:39:23.628374 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":144,"threads_started":1}
I20260928 06:39:23.630422 24551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59c8a1425b5441988caa98ddee9eafc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c8a1425b5441988caa98ddee9eafc4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 36935 } }
I20260928 06:39:23.630780 24551 leader_election.cc:302] T 00000000000000000000000000000000 P 59c8a1425b5441988caa98ddee9eafc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59c8a1425b5441988caa98ddee9eafc4; no voters:
I20260928 06:39:23.631110 24551 leader_election.cc:288] T 00000000000000000000000000000000 P 59c8a1425b5441988caa98ddee9eafc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:23.631278 24556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59c8a1425b5441988caa98ddee9eafc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:23.631526 24556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59c8a1425b5441988caa98ddee9eafc4 [term 1 LEADER]: Becoming Leader. State: Replica: 59c8a1425b5441988caa98ddee9eafc4, State: Running, Role: LEADER
I20260928 06:39:23.631868 24556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59c8a1425b5441988caa98ddee9eafc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c8a1425b5441988caa98ddee9eafc4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 36935 } }
I20260928 06:39:23.632404 24551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59c8a1425b5441988caa98ddee9eafc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:23.633612 24558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59c8a1425b5441988caa98ddee9eafc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59c8a1425b5441988caa98ddee9eafc4. Latest consensus state: current_term: 1 leader_uuid: "59c8a1425b5441988caa98ddee9eafc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c8a1425b5441988caa98ddee9eafc4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 36935 } } }
I20260928 06:39:23.633752 24558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59c8a1425b5441988caa98ddee9eafc4 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:23.634114 24557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59c8a1425b5441988caa98ddee9eafc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59c8a1425b5441988caa98ddee9eafc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c8a1425b5441988caa98ddee9eafc4" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 36935 } } }
I20260928 06:39:23.634217 24567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:23.634232 24557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59c8a1425b5441988caa98ddee9eafc4 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:23.637521 24567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:23.643329 24567 catalog_manager.cc:1347] Generated new cluster ID: d76b332ce8454d2db8c21b41ae3caa3c
I20260928 06:39:23.643422 24567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:23.655925 24567 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:23.657203 24567 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:23.662878 24567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59c8a1425b5441988caa98ddee9eafc4: Generated new TSK 0
I20260928 06:39:23.663774 24567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.22.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
Entry for principal kudu/127.5.22.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab.
I20260928 06:39:23.697970 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:36935
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:23.815924 24577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:23.816264 24577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:23.816349 24577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:23.820451 24577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:23.820544 24577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:23.820639 24577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:39:23.826026 24577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.1.keytab
--principal=kudu/127.5.22.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:36935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:23.827210 24577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:23.828300 24577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:23.838932 24582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:23.838917 24585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:23.839041 24577 server_base.cc:1034] running on GCE node
W20260928 06:39:23.838915 24583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:23.839682 24577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:23.840407 24577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:23.841631 24577 hybrid_clock.cc:648] HybridClock initialized: now 1790577563841601 us; error 69 us; skew 500 ppm
Sep 28 06:39:23 dist-test-slave-j0qf krb5kdc[24469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577563, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:23.844694 24577 init.cc:377] Logged in from keytab as kudu/127.5.22.1@KRBTEST.COM (short username kudu)
I20260928 06:39:23.845582 24577 webserver.cc:460] Webserver started at http://127.5.22.1:41357/ using document root <none> and password file <none>
I20260928 06:39:23.846124 24577 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:23.846207 24577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:23.846462 24577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:23.848515 24577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "5b71b7b63f144bd79ed9db0f76604173"
format_stamp: "Formatted at 2026-09-28 06:39:23 on dist-test-slave-j0qf"
server_key: "525a9942b914494f25f4e5886a57e627"
server_key_iv: "febe7dd1b21650f5358e0620fbaee08b"
server_key_version: "encryptionkey@0"
I20260928 06:39:23.849035 24577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "5b71b7b63f144bd79ed9db0f76604173"
format_stamp: "Formatted at 2026-09-28 06:39:23 on dist-test-slave-j0qf"
server_key: "525a9942b914494f25f4e5886a57e627"
server_key_iv: "febe7dd1b21650f5358e0620fbaee08b"
server_key_version: "encryptionkey@0"
I20260928 06:39:23.852748 24577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:39:23.855168 24592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:23.856441 24577 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:39:23.856534 24577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "5b71b7b63f144bd79ed9db0f76604173"
format_stamp: "Formatted at 2026-09-28 06:39:23 on dist-test-slave-j0qf"
server_key: "525a9942b914494f25f4e5886a57e627"
server_key_iv: "febe7dd1b21650f5358e0620fbaee08b"
server_key_version: "encryptionkey@0"
I20260928 06:39:23.856617 24577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:23.868626 24577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:23.872031 24577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:23.872205 24577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:23.872764 24577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:23.873556 24577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:23.873620 24577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:23.873660 24577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:23.873677 24577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:23.882668 24577 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:35921
I20260928 06:39:23.882761 24706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:35921 every 8 connection(s)
I20260928 06:39:23.883644 24577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:39:23.885143 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 24577
I20260928 06:39:23.885290 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:39:23.885627 5208 external_mini_cluster.cc:1428] Setting key 7870b368933e63650fdecfa2407dcc0d
Sep 28 06:39:23 dist-test-slave-j0qf krb5kdc[24469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577563, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.1@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.22.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.2@KRBTEST.COM" created.
I20260928 06:39:23.899127 24554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:23.885826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:44453 (local address 127.5.22.62:36935)
0928 06:39:23.885991 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:23.885996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:23.886971 (+ 975us) server_negotiation.cc:407] Connection header received
0928 06:39:23.888602 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:23.888606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:23.888658 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:23.888749 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:23.889947 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:23.890849 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:23.892173 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:23.892619 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:23.892765 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:23.895262 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:23.895269 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:23.895275 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:23.895301 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:23.896850 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:23.897303 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:23.897305 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:23.897307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:23.897387 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:23.897706 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:23.897708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:23.897709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:23.897920 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:23.897996 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:23.898628 (+ 632us) server_negotiation.cc:299] Negotiation successful
0928 06:39:23.898845 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 06:39:23.900059 24709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:23.886213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:36935 (local address 127.5.22.1:44453)
0928 06:39:23.886818 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:23.886877 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:23.888365 (+ 1488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:23.888924 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:23.888931 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:23.889309 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:23.889790 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:23.889804 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:23.891012 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:23.891015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:23.892018 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:23.892028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:23.892790 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:23.892793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:23.892946 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:23.893572 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:23.893588 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:23.895081 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:23.896955 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:23.896959 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:23.896971 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:23.897214 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:23.897471 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:23.897473 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:23.897474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:23.897601 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:23.898019 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:23.898024 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:23.898257 (+ 233us) client_negotiation.cc:769] Sending connection context
0928 06:39:23.898535 (+ 278us) client_negotiation.cc:240] Negotiation successful
0928 06:39:23.898890 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":146,"threads_started":1}
I20260928 06:39:23.901754 24707 heartbeater.cc:347] Connected to a master server at 127.5.22.62:36935
I20260928 06:39:23.902027 24707 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:23.902676 24707 heartbeater.cc:511] Master 127.5.22.62:36935 requested a full tablet report, sending...
I20260928 06:39:23.904433 24514 ts_manager.cc:194] Registered new tserver with Master: 5b71b7b63f144bd79ed9db0f76604173 (127.5.22.1:35921)
I20260928 06:39:23.905596 24514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.1@KRBTEST.COM'} at 127.5.22.1:44453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
Entry for principal kudu/127.5.22.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab.
I20260928 06:39:23.918709 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:36935
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:24.036656 24712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:24.036901 24712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:24.036947 24712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:24.041013 24712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:24.041074 24712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:24.041167 24712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:39:24.046550 24712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.2.keytab
--principal=kudu/127.5.22.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:36935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:24.047559 24712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:24.048568 24712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:24.058123 24720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:24.058127 24717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:24.058099 24718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:24.058256 24712 server_base.cc:1034] running on GCE node
I20260928 06:39:24.058740 24712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:24.059336 24712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:24.060526 24712 hybrid_clock.cc:648] HybridClock initialized: now 1790577564060490 us; error 45 us; skew 500 ppm
Sep 28 06:39:24 dist-test-slave-j0qf krb5kdc[24469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577564, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:24.063416 24712 init.cc:377] Logged in from keytab as kudu/127.5.22.2@KRBTEST.COM (short username kudu)
I20260928 06:39:24.064277 24712 webserver.cc:460] Webserver started at http://127.5.22.2:44115/ using document root <none> and password file <none>
I20260928 06:39:24.064918 24712 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:24.065016 24712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:24.065231 24712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:24.067462 24712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "9f784ab38db7411cbef3db68ca4ffe97"
format_stamp: "Formatted at 2026-09-28 06:39:24 on dist-test-slave-j0qf"
server_key: "0435fac56aba7391c5058aad12b3b7ab"
server_key_iv: "eb21fff9e6c8c5fed48eb2e47e5c1223"
server_key_version: "encryptionkey@0"
I20260928 06:39:24.068032 24712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "9f784ab38db7411cbef3db68ca4ffe97"
format_stamp: "Formatted at 2026-09-28 06:39:24 on dist-test-slave-j0qf"
server_key: "0435fac56aba7391c5058aad12b3b7ab"
server_key_iv: "eb21fff9e6c8c5fed48eb2e47e5c1223"
server_key_version: "encryptionkey@0"
I20260928 06:39:24.071717 24712 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 06:39:24.074185 24727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:24.075472 24712 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:39:24.075611 24712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "9f784ab38db7411cbef3db68ca4ffe97"
format_stamp: "Formatted at 2026-09-28 06:39:24 on dist-test-slave-j0qf"
server_key: "0435fac56aba7391c5058aad12b3b7ab"
server_key_iv: "eb21fff9e6c8c5fed48eb2e47e5c1223"
server_key_version: "encryptionkey@0"
I20260928 06:39:24.075727 24712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:24.098935 24712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:24.102380 24712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:24.102597 24712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:24.103209 24712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:24.104038 24712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:24.104115 24712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:24.104187 24712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:24.104221 24712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:24.113050 24712 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:34017
I20260928 06:39:24.113078 24841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:34017 every 8 connection(s)
I20260928 06:39:24.113947 24712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:39:24.115693 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 24712
I20260928 06:39:24.115823 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:39:24.116184 5208 external_mini_cluster.cc:1428] Setting key 2e1fd0ef409059bbef2fa08738999d81
Sep 28 06:39:24 dist-test-slave-j0qf krb5kdc[24469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577564, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.2@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.22.3@KRBTEST.COM; defaulting to no policy
I20260928 06:39:24.129660 24554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:24.115886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:35435 (local address 127.5.22.62:36935)
0928 06:39:24.116060 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:24.116065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:24.116951 (+ 886us) server_negotiation.cc:407] Connection header received
0928 06:39:24.118349 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:24.118353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:24.118408 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:24.118492 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:24.120006 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.120524 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.122412 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.122628 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.122707 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:24.125740 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:24.125744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:24.125746 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:24.125769 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:24.127433 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:24.127954 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:24.127957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:24.127958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:24.128036 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:24.128418 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:24.128422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:24.128425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:24.128710 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:24.128839 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:24.129425 (+ 586us) server_negotiation.cc:299] Negotiation successful
0928 06:39:24.129535 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.3@KRBTEST.COM" created.
I20260928 06:39:24.130657 24844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:24.116310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:36935 (local address 127.5.22.2:35435)
0928 06:39:24.116794 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:24.116858 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:24.118096 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:24.118687 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:24.118693 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:24.119049 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:24.119839 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:24.119863 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.120661 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.120667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:24.122229 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:24.122240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.122773 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.122779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:24.123045 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:24.123756 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:24.123784 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:24.125541 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:24.127574 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:24.127579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:24.127593 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:24.127852 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:24.128132 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:24.128135 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:24.128136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:24.128292 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:24.128890 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:24.128898 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:24.129159 (+ 261us) client_negotiation.cc:769] Sending connection context
0928 06:39:24.129355 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 06:39:24.129696 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":133,"threads_started":1}
I20260928 06:39:24.132141 24842 heartbeater.cc:347] Connected to a master server at 127.5.22.62:36935
I20260928 06:39:24.132424 24842 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:24.133046 24842 heartbeater.cc:511] Master 127.5.22.62:36935 requested a full tablet report, sending...
I20260928 06:39:24.134249 24514 ts_manager.cc:194] Registered new tserver with Master: 9f784ab38db7411cbef3db68ca4ffe97 (127.5.22.2:34017)
I20260928 06:39:24.134954 24514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.2@KRBTEST.COM'} at 127.5.22.2:35435
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
Entry for principal kudu/127.5.22.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab.
I20260928 06:39:24.150223 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:36935
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:24.270265 24847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:24.270536 24847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:24.270579 24847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:24.274549 24847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:24.274602 24847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:24.274658 24847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:39:24.280105 24847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.3.keytab
--principal=kudu/127.5.22.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:36935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:24.281409 24847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:24.282428 24847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:24.292017 24855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:24.292039 24853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:24.292006 24852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:24.292402 24847 server_base.cc:1034] running on GCE node
I20260928 06:39:24.292817 24847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:24.293370 24847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:24.294528 24847 hybrid_clock.cc:648] HybridClock initialized: now 1790577564294509 us; error 29 us; skew 500 ppm
Sep 28 06:39:24 dist-test-slave-j0qf krb5kdc[24469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577564, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:24.297343 24847 init.cc:377] Logged in from keytab as kudu/127.5.22.3@KRBTEST.COM (short username kudu)
I20260928 06:39:24.298249 24847 webserver.cc:460] Webserver started at http://127.5.22.3:44287/ using document root <none> and password file <none>
I20260928 06:39:24.298851 24847 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:24.298913 24847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:24.299150 24847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:24.301151 24847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "278215765cc1484796493defc1fe3c5c"
format_stamp: "Formatted at 2026-09-28 06:39:24 on dist-test-slave-j0qf"
server_key: "80ec3117e443bf691c3e1e6954c136ef"
server_key_iv: "7e647507ae60c4005509c6edd583bad4"
server_key_version: "encryptionkey@0"
I20260928 06:39:24.301719 24847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "278215765cc1484796493defc1fe3c5c"
format_stamp: "Formatted at 2026-09-28 06:39:24 on dist-test-slave-j0qf"
server_key: "80ec3117e443bf691c3e1e6954c136ef"
server_key_iv: "7e647507ae60c4005509c6edd583bad4"
server_key_version: "encryptionkey@0"
I20260928 06:39:24.305827 24847 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 06:39:24.308430 24862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:24.309917 24847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:39:24.310029 24847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "278215765cc1484796493defc1fe3c5c"
format_stamp: "Formatted at 2026-09-28 06:39:24 on dist-test-slave-j0qf"
server_key: "80ec3117e443bf691c3e1e6954c136ef"
server_key_iv: "7e647507ae60c4005509c6edd583bad4"
server_key_version: "encryptionkey@0"
I20260928 06:39:24.310249 24847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:24.329769 24847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:24.333899 24847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:24.334141 24847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:24.334877 24847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:24.335831 24847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:24.335884 24847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:24.335927 24847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:24.335990 24847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:24.345767 24976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:40161 every 8 connection(s)
I20260928 06:39:24.345746 24847 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:40161
I20260928 06:39:24.346912 24847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:39:24.347626 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 24847
I20260928 06:39:24.347829 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
I20260928 06:39:24.348174 5208 external_mini_cluster.cc:1428] Setting key aac61b3dce699543361434437eeb1cc5
Sep 28 06:39:24 dist-test-slave-j0qf krb5kdc[24469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577564, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.3@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:39:24.363179 24554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:24.350808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:47787 (local address 127.5.22.62:36935)
0928 06:39:24.350970 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:24.350975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:24.350992 (+ 17us) server_negotiation.cc:407] Connection header received
0928 06:39:24.351948 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:24.351951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:24.352005 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:24.352087 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:24.353486 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.354407 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.355725 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.355981 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.356114 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:24.358890 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:24.358894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:24.358896 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:24.358920 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:24.360492 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:24.361277 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:24.361283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:24.361285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:24.361380 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:24.361802 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:24.361805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:24.361806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:24.362111 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:24.362274 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:24.362936 (+ 662us) server_negotiation.cc:299] Negotiation successful
0928 06:39:24.363051 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 06:39:24.364066 24979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:24.349757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:36935 (local address 127.5.22.3:47787)
0928 06:39:24.350392 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:24.350454 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:24.351740 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:24.352307 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:24.352314 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:24.352839 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:24.353304 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:24.353318 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.354562 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.354566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:24.355553 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:24.355563 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.356154 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.356159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:24.356350 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:24.357106 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:24.357124 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:24.358700 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:24.360641 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:24.360647 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:24.360683 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:24.361101 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:24.361499 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:24.361503 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:24.361505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:24.361693 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:24.362239 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:24.362246 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:24.362619 (+ 373us) client_negotiation.cc:769] Sending connection context
0928 06:39:24.362885 (+ 266us) client_negotiation.cc:240] Negotiation successful
0928 06:39:24.363172 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":123,"threads_started":1}
I20260928 06:39:24.365447 24977 heartbeater.cc:347] Connected to a master server at 127.5.22.62:36935
I20260928 06:39:24.365741 24977 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:24.366297 24977 heartbeater.cc:511] Master 127.5.22.62:36935 requested a full tablet report, sending...
I20260928 06:39:24.367578 24514 ts_manager.cc:194] Registered new tserver with Master: 278215765cc1484796493defc1fe3c5c (127.5.22.3:40161)
I20260928 06:39:24.368247 24514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.22.3@KRBTEST.COM'} at 127.5.22.3:47787
I20260928 06:39:24.374567 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:39:24.384279 24554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:24.376528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41370 (local address 127.5.22.62:36935)
0928 06:39:24.376666 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:24.376672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:24.376862 (+ 190us) server_negotiation.cc:407] Connection header received
0928 06:39:24.377041 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:24.377044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:24.377091 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:24.377183 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:24.377909 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.378615 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.379601 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.379910 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.379992 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:24.381376 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:24.381379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:24.381381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:24.381401 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:24.382874 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:24.383253 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:24.383256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:24.383257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:24.383327 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:24.383582 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:24.383584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:24.383585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:24.383782 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:24.383884 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:24.384078 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 06:39:24.384166 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 06:39:24.387249 24514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:39:24.390374 24514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:39:24.404872 24990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:24.398849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:34017 (local address 127.0.0.1:41410)
0928 06:39:24.399712 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:24.399766 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:24.399908 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:24.400904 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:24.400910 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:24.400931 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:24.401094 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:24.401102 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.402242 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.402245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:24.403436 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:24.403445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.404250 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.404254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:24.404361 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:24.404399 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 06:39:24.404551 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 06:39:24.404714 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":116,"threads_started":1}
I20260928 06:39:24.404873 24993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:24.400049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41410 (local address 127.5.22.2:34017)
0928 06:39:24.400397 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:24.400406 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:24.400426 (+ 20us) server_negotiation.cc:407] Connection header received
0928 06:39:24.400519 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:24.400524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:24.400728 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:24.400902 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:24.401216 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.402086 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.403569 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.404116 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.404252 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:24.404418 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:24.404585 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 06:39:24.404683 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":108,"threads_started":1}
I20260928 06:39:24.406086 24988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:24.398965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:40161 (local address 127.0.0.1:52278)
0928 06:39:24.400127 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:24.400145 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:24.400220 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:24.401123 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:24.401125 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:24.401136 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:24.401262 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:24.401269 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.402742 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.402747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:24.404173 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:24.404184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.405676 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.405679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:24.405786 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:24.405808 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 06:39:24.405873 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:39:24.405943 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1096,"mutex_wait_us":189,"thread_start_us":105,"threads_started":1}
I20260928 06:39:24.406401 24992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:24.399681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52278 (local address 127.5.22.3:40161)
0928 06:39:24.400472 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:24.400674 (+ 202us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:24.400696 (+ 22us) server_negotiation.cc:407] Connection header received
0928 06:39:24.400757 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:24.400761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:24.400973 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:24.401139 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:24.401383 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.402581 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.404341 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.405557 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.405748 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:24.405906 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:24.405953 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 06:39:24.406035 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":104,"threads_started":1}
I20260928 06:39:24.406802 24991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:24.399462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47092 (local address 127.5.22.1:35921)
0928 06:39:24.400002 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:24.400021 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:24.400542 (+ 521us) server_negotiation.cc:407] Connection header received
0928 06:39:24.400624 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:24.400628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:24.400771 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:24.400910 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:24.401526 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.402663 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.404320 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.405256 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.405352 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:24.405486 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:24.406562 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0928 06:39:24.406668 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":88,"threads_started":1}
I20260928 06:39:24.407208 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:24.399240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:35921 (local address 127.0.0.1:47092)
0928 06:39:24.400427 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:24.400439 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:24.400531 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:24.401069 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:24.401072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:24.401094 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:24.401406 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:24.401412 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.402778 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.402781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:24.404173 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:24.404184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.405600 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.405607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:24.406384 (+ 777us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:24.406413 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 06:39:24.406994 (+ 581us) client_negotiation.cc:240] Negotiation successful
0928 06:39:24.407072 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1131,"mutex_wait_us":123,"thread_start_us":93,"threads_started":1}
I20260928 06:39:24.408280 24775 tablet_service.cc:1467] Processing CreateTablet for tablet 9c59ebb0b276404599bf20da78b46fdf (DEFAULT_TABLE table=test-table [id=4f414feef0044582a2fe4091632b3309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:24.409444 24775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c59ebb0b276404599bf20da78b46fdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:24.409066 24910 tablet_service.cc:1467] Processing CreateTablet for tablet 9c59ebb0b276404599bf20da78b46fdf (DEFAULT_TABLE table=test-table [id=4f414feef0044582a2fe4091632b3309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:24.409332 24640 tablet_service.cc:1467] Processing CreateTablet for tablet 9c59ebb0b276404599bf20da78b46fdf (DEFAULT_TABLE table=test-table [id=4f414feef0044582a2fe4091632b3309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:24.410089 24910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c59ebb0b276404599bf20da78b46fdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:24.410380 24640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c59ebb0b276404599bf20da78b46fdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:24.416421 24994 tablet_bootstrap.cc:492] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173: Bootstrap starting.
I20260928 06:39:24.417692 24995 tablet_bootstrap.cc:492] T 9c59ebb0b276404599bf20da78b46fdf P 9f784ab38db7411cbef3db68ca4ffe97: Bootstrap starting.
I20260928 06:39:24.418680 24994 tablet_bootstrap.cc:654] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:24.418684 24996 tablet_bootstrap.cc:492] T 9c59ebb0b276404599bf20da78b46fdf P 278215765cc1484796493defc1fe3c5c: Bootstrap starting.
I20260928 06:39:24.419425 24994 log.cc:824] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:24.419788 24995 tablet_bootstrap.cc:654] T 9c59ebb0b276404599bf20da78b46fdf P 9f784ab38db7411cbef3db68ca4ffe97: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:24.420750 24996 tablet_bootstrap.cc:654] T 9c59ebb0b276404599bf20da78b46fdf P 278215765cc1484796493defc1fe3c5c: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:24.421231 24995 log.cc:824] T 9c59ebb0b276404599bf20da78b46fdf P 9f784ab38db7411cbef3db68ca4ffe97: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:24.421578 24994 tablet_bootstrap.cc:492] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173: No bootstrap required, opened a new log
I20260928 06:39:24.421798 24994 ts_tablet_manager.cc:1397] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 06:39:24.421972 24996 log.cc:824] T 9c59ebb0b276404599bf20da78b46fdf P 278215765cc1484796493defc1fe3c5c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:24.423364 24995 tablet_bootstrap.cc:492] T 9c59ebb0b276404599bf20da78b46fdf P 9f784ab38db7411cbef3db68ca4ffe97: No bootstrap required, opened a new log
I20260928 06:39:24.423568 24995 ts_tablet_manager.cc:1397] T 9c59ebb0b276404599bf20da78b46fdf P 9f784ab38db7411cbef3db68ca4ffe97: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 06:39:24.423733 24996 tablet_bootstrap.cc:492] T 9c59ebb0b276404599bf20da78b46fdf P 278215765cc1484796493defc1fe3c5c: No bootstrap required, opened a new log
I20260928 06:39:24.423925 24996 ts_tablet_manager.cc:1397] T 9c59ebb0b276404599bf20da78b46fdf P 278215765cc1484796493defc1fe3c5c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 06:39:24.426497 24994 raft_consensus.cc:348] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f784ab38db7411cbef3db68ca4ffe97" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34017 } } peers { permanent_uuid: "278215765cc1484796493defc1fe3c5c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40161 } } peers { permanent_uuid: "5b71b7b63f144bd79ed9db0f76604173" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 35921 } }
I20260928 06:39:24.426800 24994 raft_consensus.cc:374] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:24.426961 24994 raft_consensus.cc:729] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b71b7b63f144bd79ed9db0f76604173, State: Initialized, Role: FOLLOWER
I20260928 06:39:24.427217 24996 raft_consensus.cc:348] T 9c59ebb0b276404599bf20da78b46fdf P 278215765cc1484796493defc1fe3c5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f784ab38db7411cbef3db68ca4ffe97" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34017 } } peers { permanent_uuid: "278215765cc1484796493defc1fe3c5c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40161 } } peers { permanent_uuid: "5b71b7b63f144bd79ed9db0f76604173" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 35921 } }
I20260928 06:39:24.427534 24996 raft_consensus.cc:374] T 9c59ebb0b276404599bf20da78b46fdf P 278215765cc1484796493defc1fe3c5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:24.427600 24996 raft_consensus.cc:729] T 9c59ebb0b276404599bf20da78b46fdf P 278215765cc1484796493defc1fe3c5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 278215765cc1484796493defc1fe3c5c, State: Initialized, Role: FOLLOWER
I20260928 06:39:24.427523 24994 consensus_queue.cc:260] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f784ab38db7411cbef3db68ca4ffe97" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34017 } } peers { permanent_uuid: "278215765cc1484796493defc1fe3c5c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40161 } } peers { permanent_uuid: "5b71b7b63f144bd79ed9db0f76604173" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 35921 } }
I20260928 06:39:24.428071 24996 consensus_queue.cc:260] T 9c59ebb0b276404599bf20da78b46fdf P 278215765cc1484796493defc1fe3c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f784ab38db7411cbef3db68ca4ffe97" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34017 } } peers { permanent_uuid: "278215765cc1484796493defc1fe3c5c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40161 } } peers { permanent_uuid: "5b71b7b63f144bd79ed9db0f76604173" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 35921 } }
I20260928 06:39:24.428642 24994 ts_tablet_manager.cc:1428] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 06:39:24.428773 24977 heartbeater.cc:503] Master 127.5.22.62:36935 was elected leader, sending a full tablet report...
I20260928 06:39:24.428843 24995 raft_consensus.cc:348] T 9c59ebb0b276404599bf20da78b46fdf P 9f784ab38db7411cbef3db68ca4ffe97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f784ab38db7411cbef3db68ca4ffe97" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34017 } } peers { permanent_uuid: "278215765cc1484796493defc1fe3c5c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40161 } } peers { permanent_uuid: "5b71b7b63f144bd79ed9db0f76604173" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 35921 } }
I20260928 06:39:24.429065 24996 ts_tablet_manager.cc:1428] T 9c59ebb0b276404599bf20da78b46fdf P 278215765cc1484796493defc1fe3c5c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 06:39:24.429097 24995 raft_consensus.cc:374] T 9c59ebb0b276404599bf20da78b46fdf P 9f784ab38db7411cbef3db68ca4ffe97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:24.429149 24995 raft_consensus.cc:729] T 9c59ebb0b276404599bf20da78b46fdf P 9f784ab38db7411cbef3db68ca4ffe97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f784ab38db7411cbef3db68ca4ffe97, State: Initialized, Role: FOLLOWER
I20260928 06:39:24.429221 24707 heartbeater.cc:503] Master 127.5.22.62:36935 was elected leader, sending a full tablet report...
I20260928 06:39:24.429579 24995 consensus_queue.cc:260] T 9c59ebb0b276404599bf20da78b46fdf P 9f784ab38db7411cbef3db68ca4ffe97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f784ab38db7411cbef3db68ca4ffe97" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34017 } } peers { permanent_uuid: "278215765cc1484796493defc1fe3c5c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40161 } } peers { permanent_uuid: "5b71b7b63f144bd79ed9db0f76604173" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 35921 } }
I20260928 06:39:24.430223 24842 heartbeater.cc:503] Master 127.5.22.62:36935 was elected leader, sending a full tablet report...
I20260928 06:39:24.432071 24995 ts_tablet_manager.cc:1428] T 9c59ebb0b276404599bf20da78b46fdf P 9f784ab38db7411cbef3db68ca4ffe97: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260928 06:39:24.597965 24978 tablet.cc:2369] T 9c59ebb0b276404599bf20da78b46fdf P 278215765cc1484796493defc1fe3c5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:39:24.615231 24843 tablet.cc:2369] T 9c59ebb0b276404599bf20da78b46fdf P 9f784ab38db7411cbef3db68ca4ffe97: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:39:24.635115 24708 tablet.cc:2369] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:39:24.737749 25000 raft_consensus.cc:482] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:39:24.738008 25000 raft_consensus.cc:504] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f784ab38db7411cbef3db68ca4ffe97" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34017 } } peers { permanent_uuid: "278215765cc1484796493defc1fe3c5c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40161 } } peers { permanent_uuid: "5b71b7b63f144bd79ed9db0f76604173" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 35921 } }
I20260928 06:39:24.739287 25000 leader_election.cc:288] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f784ab38db7411cbef3db68ca4ffe97 (127.5.22.2:34017), 278215765cc1484796493defc1fe3c5c (127.5.22.3:40161)
I20260928 06:39:24.743574 24993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:24.740038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:48969 (local address 127.5.22.2:34017)
0928 06:39:24.740202 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:24.740206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:24.740228 (+ 22us) server_negotiation.cc:407] Connection header received
0928 06:39:24.740289 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:24.740295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:24.740342 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:24.740414 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:24.740770 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.741299 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.742611 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.742990 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.743096 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:24.743175 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:24.743355 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 06:39:24.743437 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 06:39:24.743608 24992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:24.739748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:60835 (local address 127.5.22.3:40161)
0928 06:39:24.739912 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:24.739917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:24.740191 (+ 274us) server_negotiation.cc:407] Connection header received
0928 06:39:24.740325 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:24.740327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:24.740373 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:24.740477 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:24.740844 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.741326 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.742546 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.743003 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.743128 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:24.743193 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:24.743427 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 06:39:24.743505 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 06:39:24.743602 25003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:24.739613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:34017 (local address 127.5.22.1:48969)
0928 06:39:24.739970 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:24.739986 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:24.740103 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:24.740457 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:24.740463 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:24.740483 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:24.740651 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:24.740657 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.741414 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.741419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:24.742462 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:24.742472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.743114 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.743118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:24.743222 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:24.743246 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 06:39:24.743324 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:39:24.743423 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":82,"thread_start_us":155,"threads_started":1}
I20260928 06:39:24.743610 25004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:24.739613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:40161 (local address 127.5.22.1:60835)
0928 06:39:24.740069 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:24.740102 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:24.740231 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:24.740505 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:24.740511 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:24.740538 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:24.740671 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:24.740675 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.741469 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.741471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:24.742415 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:24.742425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.743151 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.743155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:24.743272 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:24.743290 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 06:39:24.743388 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 06:39:24.743454 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":52,"spinlock_wait_cycles":768,"thread_start_us":143,"threads_started":1}
I20260928 06:39:24.744488 24930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c59ebb0b276404599bf20da78b46fdf" candidate_uuid: "5b71b7b63f144bd79ed9db0f76604173" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "278215765cc1484796493defc1fe3c5c" is_pre_election: true
I20260928 06:39:24.744488 24795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c59ebb0b276404599bf20da78b46fdf" candidate_uuid: "5b71b7b63f144bd79ed9db0f76604173" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f784ab38db7411cbef3db68ca4ffe97" is_pre_election: true
I20260928 06:39:24.744830 24930 raft_consensus.cc:2454] T 9c59ebb0b276404599bf20da78b46fdf P 278215765cc1484796493defc1fe3c5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b71b7b63f144bd79ed9db0f76604173 in term 0.
I20260928 06:39:24.744830 24795 raft_consensus.cc:2454] T 9c59ebb0b276404599bf20da78b46fdf P 9f784ab38db7411cbef3db68ca4ffe97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b71b7b63f144bd79ed9db0f76604173 in term 0.
I20260928 06:39:24.745389 24596 leader_election.cc:302] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 278215765cc1484796493defc1fe3c5c, 5b71b7b63f144bd79ed9db0f76604173; no voters:
I20260928 06:39:24.745682 25000 raft_consensus.cc:2781] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:39:24.745790 25000 raft_consensus.cc:482] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:39:24.745873 25000 raft_consensus.cc:3037] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:24.747427 25000 raft_consensus.cc:504] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f784ab38db7411cbef3db68ca4ffe97" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34017 } } peers { permanent_uuid: "278215765cc1484796493defc1fe3c5c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40161 } } peers { permanent_uuid: "5b71b7b63f144bd79ed9db0f76604173" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 35921 } }
I20260928 06:39:24.747826 25000 leader_election.cc:288] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173 [CANDIDATE]: Term 1 election: Requested vote from peers 9f784ab38db7411cbef3db68ca4ffe97 (127.5.22.2:34017), 278215765cc1484796493defc1fe3c5c (127.5.22.3:40161)
I20260928 06:39:24.748286 24930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c59ebb0b276404599bf20da78b46fdf" candidate_uuid: "5b71b7b63f144bd79ed9db0f76604173" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "278215765cc1484796493defc1fe3c5c"
I20260928 06:39:24.748382 24795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c59ebb0b276404599bf20da78b46fdf" candidate_uuid: "5b71b7b63f144bd79ed9db0f76604173" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f784ab38db7411cbef3db68ca4ffe97"
I20260928 06:39:24.748445 24930 raft_consensus.cc:3037] T 9c59ebb0b276404599bf20da78b46fdf P 278215765cc1484796493defc1fe3c5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:24.748508 24795 raft_consensus.cc:3037] T 9c59ebb0b276404599bf20da78b46fdf P 9f784ab38db7411cbef3db68ca4ffe97 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:24.749933 24930 raft_consensus.cc:2454] T 9c59ebb0b276404599bf20da78b46fdf P 278215765cc1484796493defc1fe3c5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b71b7b63f144bd79ed9db0f76604173 in term 1.
I20260928 06:39:24.749933 24795 raft_consensus.cc:2454] T 9c59ebb0b276404599bf20da78b46fdf P 9f784ab38db7411cbef3db68ca4ffe97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b71b7b63f144bd79ed9db0f76604173 in term 1.
I20260928 06:39:24.750284 24596 leader_election.cc:302] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 278215765cc1484796493defc1fe3c5c, 5b71b7b63f144bd79ed9db0f76604173; no voters:
I20260928 06:39:24.750519 25000 raft_consensus.cc:2781] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:24.750762 25000 raft_consensus.cc:686] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173 [term 1 LEADER]: Becoming Leader. State: Replica: 5b71b7b63f144bd79ed9db0f76604173, State: Running, Role: LEADER
I20260928 06:39:24.751106 25000 consensus_queue.cc:237] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f784ab38db7411cbef3db68ca4ffe97" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34017 } } peers { permanent_uuid: "278215765cc1484796493defc1fe3c5c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40161 } } peers { permanent_uuid: "5b71b7b63f144bd79ed9db0f76604173" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 35921 } }
I20260928 06:39:24.754673 24513 catalog_manager.cc:5495] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b71b7b63f144bd79ed9db0f76604173 (127.5.22.1). New cstate: current_term: 1 leader_uuid: "5b71b7b63f144bd79ed9db0f76604173" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f784ab38db7411cbef3db68ca4ffe97" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "278215765cc1484796493defc1fe3c5c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b71b7b63f144bd79ed9db0f76604173" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 35921 } health_report { overall_health: HEALTHY } } }
I20260928 06:39:24.767189 24991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:24.763463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47104 (local address 127.5.22.1:35921)
0928 06:39:24.763586 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:24.763591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:24.763609 (+ 18us) server_negotiation.cc:407] Connection header received
0928 06:39:24.763754 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:24.763757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:24.763813 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:24.763905 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:24.764141 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.764699 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.765692 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:24.765903 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:24.765981 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:24.766293 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:24.766792 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:24.766895 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:24.767011 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 06:39:24.767063 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:39:24.773361 24795 raft_consensus.cc:1261] T 9c59ebb0b276404599bf20da78b46fdf P 9f784ab38db7411cbef3db68ca4ffe97 [term 1 FOLLOWER]: Refusing update from remote peer 5b71b7b63f144bd79ed9db0f76604173: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:24.773361 24930 raft_consensus.cc:1261] T 9c59ebb0b276404599bf20da78b46fdf P 278215765cc1484796493defc1fe3c5c [term 1 FOLLOWER]: Refusing update from remote peer 5b71b7b63f144bd79ed9db0f76604173: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:24.774173 25005 consensus_queue.cc:1035] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173 [LEADER]: Connected to new peer: Peer: permanent_uuid: "278215765cc1484796493defc1fe3c5c" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 40161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:24.774441 25000 consensus_queue.cc:1035] T 9c59ebb0b276404599bf20da78b46fdf P 5b71b7b63f144bd79ed9db0f76604173 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f784ab38db7411cbef3db68ca4ffe97" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 34017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:24.785452 25010 mvcc.cc:205] Tried to move back new op lower bound from 7334205705303719936 to 7334205705223188480. Current Snapshot: MvccSnapshot[applied={T|T < 7334205705303719936}]
I20260928 06:39:24.791234 24513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:39:24.791400 24513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:39:24.794162 24513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59c8a1425b5441988caa98ddee9eafc4: Sending DeleteTablet for 3 replicas of tablet 9c59ebb0b276404599bf20da78b46fdf
I20260928 06:39:24.795194 24910 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c59ebb0b276404599bf20da78b46fdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:24 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:52278
I20260928 06:39:24.795334 24775 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c59ebb0b276404599bf20da78b46fdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:24 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:41410
I20260928 06:39:24.795914 25018 tablet_replica.cc:330] T 9c59ebb0b276404599bf20da78b46fdf P 278215765cc1484796493defc1fe3c5c: stopping tablet replica
I20260928 06:39:24.796155 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 24577
I20260928 06:39:24.796175 24640 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c59ebb0b276404599bf20da78b46fdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:24 UTC) from {username='kudu', principal='kudu/127.5.22.62@KRBTEST.COM'} at 127.0.0.1:47092
I20260928 06:39:24.796283 25018 raft_consensus.cc:2229] T 9c59ebb0b276404599bf20da78b46fdf P 278215765cc1484796493defc1fe3c5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:39:24.796594 25018 raft_consensus.cc:2258] T 9c59ebb0b276404599bf20da78b46fdf P 278215765cc1484796493defc1fe3c5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:24.796973 25019 tablet_replica.cc:330] T 9c59ebb0b276404599bf20da78b46fdf P 9f784ab38db7411cbef3db68ca4ffe97: stopping tablet replica
I20260928 06:39:24.797274 25019 raft_consensus.cc:2229] T 9c59ebb0b276404599bf20da78b46fdf P 9f784ab38db7411cbef3db68ca4ffe97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:39:24.797574 25019 raft_consensus.cc:2258] T 9c59ebb0b276404599bf20da78b46fdf P 9f784ab38db7411cbef3db68ca4ffe97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:24.798636 25018 ts_tablet_manager.cc:1905] T 9c59ebb0b276404599bf20da78b46fdf P 278215765cc1484796493defc1fe3c5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:39:24.800913 25019 ts_tablet_manager.cc:1905] T 9c59ebb0b276404599bf20da78b46fdf P 9f784ab38db7411cbef3db68ca4ffe97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:39:24.801724 25018 ts_tablet_manager.cc:1918] T 9c59ebb0b276404599bf20da78b46fdf P 278215765cc1484796493defc1fe3c5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:39:24.801827 25018 log.cc:1196] T 9c59ebb0b276404599bf20da78b46fdf P 278215765cc1484796493defc1fe3c5c: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/9c59ebb0b276404599bf20da78b46fdf
I20260928 06:39:24.802171 25018 ts_tablet_manager.cc:1939] T 9c59ebb0b276404599bf20da78b46fdf P 278215765cc1484796493defc1fe3c5c: Deleting consensus metadata
W20260928 06:39:24.803697 24498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.22.1:35921 (error 108)
I20260928 06:39:24.803731 25019 ts_tablet_manager.cc:1918] T 9c59ebb0b276404599bf20da78b46fdf P 9f784ab38db7411cbef3db68ca4ffe97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:39:24.803825 25019 log.cc:1196] T 9c59ebb0b276404599bf20da78b46fdf P 9f784ab38db7411cbef3db68ca4ffe97: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577308481727-5208-0/minicluster-data/ts-1/wal/wals/9c59ebb0b276404599bf20da78b46fdf
I20260928 06:39:24.803833 24501 catalog_manager.cc:4841] TS 278215765cc1484796493defc1fe3c5c (127.5.22.3:40161): tablet 9c59ebb0b276404599bf20da78b46fdf (table test-table [id=4f414feef0044582a2fe4091632b3309]) successfully deleted
I20260928 06:39:24.804234 25019 ts_tablet_manager.cc:1939] T 9c59ebb0b276404599bf20da78b46fdf P 9f784ab38db7411cbef3db68ca4ffe97: Deleting consensus metadata
I20260928 06:39:24.804802 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:24.804483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:35921 (local address 127.0.0.1:47118)
0928 06:39:24.804616 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:24.804696 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.1:35921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 06:39:24.804940 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 24712
W20260928 06:39:24.805027 24498 catalog_manager.cc:4584] TS 5b71b7b63f144bd79ed9db0f76604173 (127.5.22.1:35921): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c59ebb0b276404599bf20da78b46fdf: Network error: Client connection negotiation failed: client connection to 127.5.22.1:35921: connect: Connection refused (error 111)
I20260928 06:39:24.811753 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 24847
I20260928 06:39:24.818101 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 24482
I20260928 06:39:24.818115 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:24.817822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:34017 (local address 127.0.0.1:41424)
0928 06:39:24.817976 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:24.818029 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.2:34017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
2026-09-28T06:39:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2575 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T06:39:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:24Z Disabled control of system clock
I20260928 06:39:24.848945 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:41151
--webserver_interface=127.5.22.62
--webserver_port=0
--builtin_ntp_servers=127.5.22.20:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:41151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:39:24.961581 25024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:24.961853 25024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:24.961890 25024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:24.966010 25024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:24.966077 25024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:24.966092 25024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:24.966109 25024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:24.966125 25024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:24.974449 25024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:41151
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:41151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:24.975556 25024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:24.976928 25024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:24.985807 25030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:24.985872 25032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:24.985818 25029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:24.986192 25024 server_base.cc:1034] running on GCE node
I20260928 06:39:24.986874 25024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:24.988472 25024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:24.989692 25024 hybrid_clock.cc:648] HybridClock initialized: now 1790577564989667 us; error 46 us; skew 500 ppm
I20260928 06:39:24.991688 25024 webserver.cc:460] Webserver started at http://127.5.22.62:41399/ using document root <none> and password file <none>
I20260928 06:39:24.992300 25024 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:24.992455 25024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:24.992695 25024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:24.994962 25024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "3bf850441ba0454ea76d4980962e2740"
format_stamp: "Formatted at 2026-09-28 06:39:24 on dist-test-slave-j0qf"
server_key: "e6231ae9b40cb0141b161ddd26e415ad"
server_key_iv: "347a3e33a78f4eeb9078cbfcd8bbbbc6"
server_key_version: "encryptionkey@0"
I20260928 06:39:24.995591 25024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "3bf850441ba0454ea76d4980962e2740"
format_stamp: "Formatted at 2026-09-28 06:39:24 on dist-test-slave-j0qf"
server_key: "e6231ae9b40cb0141b161ddd26e415ad"
server_key_iv: "347a3e33a78f4eeb9078cbfcd8bbbbc6"
server_key_version: "encryptionkey@0"
I20260928 06:39:24.999404 25024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 06:39:25.001891 25038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:25.003261 25024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260928 06:39:25.003432 25024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "3bf850441ba0454ea76d4980962e2740"
format_stamp: "Formatted at 2026-09-28 06:39:24 on dist-test-slave-j0qf"
server_key: "e6231ae9b40cb0141b161ddd26e415ad"
server_key_iv: "347a3e33a78f4eeb9078cbfcd8bbbbc6"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.003554 25024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:25.036476 25024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:25.037220 25024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:25.037441 25024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:25.044734 25024 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:41151
I20260928 06:39:25.044826 25090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:41151 every 8 connection(s)
I20260928 06:39:25.045775 25024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:25.048926 25091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:25.054519 25091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bf850441ba0454ea76d4980962e2740: Bootstrap starting.
I20260928 06:39:25.055215 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 25024
I20260928 06:39:25.055382 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:25.055686 5208 external_mini_cluster.cc:1428] Setting key cc0930c39e269a3e313c37f70cce3f87
I20260928 06:39:25.057396 25091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bf850441ba0454ea76d4980962e2740: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:25.058169 25091 log.cc:824] T 00000000000000000000000000000000 P 3bf850441ba0454ea76d4980962e2740: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:25.060076 25091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bf850441ba0454ea76d4980962e2740: No bootstrap required, opened a new log
I20260928 06:39:25.063746 25091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bf850441ba0454ea76d4980962e2740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bf850441ba0454ea76d4980962e2740" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 41151 } }
I20260928 06:39:25.063993 25091 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bf850441ba0454ea76d4980962e2740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:25.064077 25091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bf850441ba0454ea76d4980962e2740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bf850441ba0454ea76d4980962e2740, State: Initialized, Role: FOLLOWER
I20260928 06:39:25.064522 25091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bf850441ba0454ea76d4980962e2740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bf850441ba0454ea76d4980962e2740" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 41151 } }
I20260928 06:39:25.064674 25091 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bf850441ba0454ea76d4980962e2740 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:25.064785 25091 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bf850441ba0454ea76d4980962e2740 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:25.064904 25091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bf850441ba0454ea76d4980962e2740 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:25.065830 25091 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bf850441ba0454ea76d4980962e2740 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bf850441ba0454ea76d4980962e2740" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 41151 } }
I20260928 06:39:25.066176 25091 leader_election.cc:302] T 00000000000000000000000000000000 P 3bf850441ba0454ea76d4980962e2740 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bf850441ba0454ea76d4980962e2740; no voters:
I20260928 06:39:25.066565 25091 leader_election.cc:288] T 00000000000000000000000000000000 P 3bf850441ba0454ea76d4980962e2740 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:25.066665 25096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bf850441ba0454ea76d4980962e2740 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:25.067160 25096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bf850441ba0454ea76d4980962e2740 [term 1 LEADER]: Becoming Leader. State: Replica: 3bf850441ba0454ea76d4980962e2740, State: Running, Role: LEADER
I20260928 06:39:25.067533 25096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bf850441ba0454ea76d4980962e2740 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bf850441ba0454ea76d4980962e2740" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 41151 } }
I20260928 06:39:25.067910 25091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bf850441ba0454ea76d4980962e2740 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:25.069191 25098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bf850441ba0454ea76d4980962e2740 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bf850441ba0454ea76d4980962e2740. Latest consensus state: current_term: 1 leader_uuid: "3bf850441ba0454ea76d4980962e2740" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bf850441ba0454ea76d4980962e2740" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 41151 } } }
I20260928 06:39:25.069330 25098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bf850441ba0454ea76d4980962e2740 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:25.069696 25105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:25.069638 25097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bf850441ba0454ea76d4980962e2740 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bf850441ba0454ea76d4980962e2740" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bf850441ba0454ea76d4980962e2740" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 41151 } } }
I20260928 06:39:25.069818 25097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bf850441ba0454ea76d4980962e2740 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:25.069626 25094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.056848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46434 (local address 127.5.22.62:41151)
0928 06:39:25.057539 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:25.057559 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:25.057601 (+ 42us) server_negotiation.cc:407] Connection header received
0928 06:39:25.058240 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:25.058257 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:25.058722 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:25.059062 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:25.059485 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.060545 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.061481 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.061972 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.062143 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.066992 (+ 4849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:25.067001 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:25.067076 (+ 75us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:25.067692 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:25.067764 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:25.067789 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:25.067898 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:25.068367 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 06:39:25.068650 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":127,"threads_started":1}
I20260928 06:39:25.072882 25105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:25.077879 25105 catalog_manager.cc:1347] Generated new cluster ID: 64f4feedfdc84772a8a0fcbbc486c686
I20260928 06:39:25.077946 25105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:25.091450 25105 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:25.092957 25105 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:25.100977 25105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bf850441ba0454ea76d4980962e2740: Generated new TSK 0
I20260928 06:39:25.101748 25105 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:25.106048 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.1:0
--local_ip_for_outbound_sockets=127.5.22.1
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:41151
--builtin_ntp_servers=127.5.22.20:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:39:25.221968 25115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:25.222226 25115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:25.222275 25115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:25.226717 25115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:25.226810 25115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:25.226917 25115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.1
I20260928 06:39:25.232010 25115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.22.1
--webserver_port=0
--tserver_master_addrs=127.5.22.62:41151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.1
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:25.233233 25115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:25.234831 25115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:25.245115 25121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:25.245169 25124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:25.245417 25122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:25.245703 25115 server_base.cc:1034] running on GCE node
I20260928 06:39:25.246170 25115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:25.246898 25115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:25.248113 25115 hybrid_clock.cc:648] HybridClock initialized: now 1790577565248069 us; error 59 us; skew 500 ppm
I20260928 06:39:25.250177 25115 webserver.cc:460] Webserver started at http://127.5.22.1:44033/ using document root <none> and password file <none>
I20260928 06:39:25.250852 25115 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:25.250954 25115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:25.251194 25115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:25.253302 25115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-0/data/instance:
uuid: "5154a49f198746aeb1a55d070ad57444"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-j0qf"
server_key: "ed0f7fafbe1953cdc94c96ddb81602f2"
server_key_iv: "39a3e900b79161b578c07c98dbb641e4"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.253904 25115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance:
uuid: "5154a49f198746aeb1a55d070ad57444"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-j0qf"
server_key: "ed0f7fafbe1953cdc94c96ddb81602f2"
server_key_iv: "39a3e900b79161b578c07c98dbb641e4"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.257740 25115 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 06:39:25.260473 25130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:25.261744 25115 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 06:39:25.261909 25115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-0/wal
uuid: "5154a49f198746aeb1a55d070ad57444"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-j0qf"
server_key: "ed0f7fafbe1953cdc94c96ddb81602f2"
server_key_iv: "39a3e900b79161b578c07c98dbb641e4"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.262034 25115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:25.280108 25115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:25.280829 25115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:25.281035 25115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:25.281646 25115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:25.282644 25115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:25.282721 25115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:25.282790 25115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:25.282841 25115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:25.292071 25115 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.1:44447
I20260928 06:39:25.292148 25245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.1:44447 every 8 connection(s)
I20260928 06:39:25.293021 25115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-0/data/info.pb
I20260928 06:39:25.293277 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 25115
I20260928 06:39:25.293401 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-0/wal/instance
I20260928 06:39:25.293767 5208 external_mini_cluster.cc:1428] Setting key c7255585943379e7e366bcf7923c28d8
I20260928 06:39:25.296150 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.2:0
--local_ip_for_outbound_sockets=127.5.22.2
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:41151
--builtin_ntp_servers=127.5.22.20:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:39:25.303292 25094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.295202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.1:35269 (local address 127.5.22.62:41151)
0928 06:39:25.295401 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:25.295406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:25.296617 (+ 1211us) server_negotiation.cc:407] Connection header received
0928 06:39:25.298350 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:25.298374 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:25.298431 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:25.298521 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:25.299539 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.300122 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.301288 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.301534 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.301653 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.302767 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:25.302778 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:25.302781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:25.302794 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:25.302821 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:25.302830 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:25.302921 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:25.303086 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 06:39:25.303148 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:39:25.304170 25248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.295724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:41151 (local address 127.5.22.1:35269)
0928 06:39:25.296451 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:25.296503 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:25.298081 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:25.298745 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:25.298757 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:25.298864 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:25.299396 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.299408 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.300263 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.300266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.301157 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.301166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.301692 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.301696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.301865 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.302497 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:39:25.302499 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:25.302538 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:39:25.302597 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:25.302923 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:25.302935 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:25.302943 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 06:39:25.303074 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 06:39:25.303359 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":147,"threads_started":1}
I20260928 06:39:25.305485 25246 heartbeater.cc:347] Connected to a master server at 127.5.22.62:41151
I20260928 06:39:25.305740 25246 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:25.306270 25246 heartbeater.cc:511] Master 127.5.22.62:41151 requested a full tablet report, sending...
I20260928 06:39:25.307966 25055 ts_manager.cc:194] Registered new tserver with Master: 5154a49f198746aeb1a55d070ad57444 (127.5.22.1:44447)
I20260928 06:39:25.309546 25055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.22.1:35269
W20260928 06:39:25.408981 25249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:25.409263 25249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:25.409301 25249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:25.413244 25249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:25.413295 25249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:25.413349 25249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.2
I20260928 06:39:25.417959 25249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.22.2
--webserver_port=0
--tserver_master_addrs=127.5.22.62:41151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.2
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:25.418988 25249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:25.420300 25249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:25.429983 25257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:25.430004 25254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:25.429999 25255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:25.430080 25249 server_base.cc:1034] running on GCE node
I20260928 06:39:25.430734 25249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:25.431383 25249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:25.432577 25249 hybrid_clock.cc:648] HybridClock initialized: now 1790577565432551 us; error 46 us; skew 500 ppm
I20260928 06:39:25.434468 25249 webserver.cc:460] Webserver started at http://127.5.22.2:45023/ using document root <none> and password file <none>
I20260928 06:39:25.435142 25249 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:25.435241 25249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:25.435452 25249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:25.437474 25249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-1/data/instance:
uuid: "6775b101cc9544d9ac1ff8c586f8432c"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-j0qf"
server_key: "9418444c803065441a3741777c632c54"
server_key_iv: "e30616085e742f0b9be7448824cbf54a"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.438050 25249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance:
uuid: "6775b101cc9544d9ac1ff8c586f8432c"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-j0qf"
server_key: "9418444c803065441a3741777c632c54"
server_key_iv: "e30616085e742f0b9be7448824cbf54a"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.441738 25249 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:39:25.444275 25263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:25.445600 25249 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 06:39:25.445832 25249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-1/wal
uuid: "6775b101cc9544d9ac1ff8c586f8432c"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-j0qf"
server_key: "9418444c803065441a3741777c632c54"
server_key_iv: "e30616085e742f0b9be7448824cbf54a"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.445969 25249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:25.477219 25249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:25.478005 25249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:25.478226 25249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:25.478899 25249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:25.479893 25249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:25.479970 25249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:25.480042 25249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:25.480098 25249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:25.489420 25249 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.2:41093
I20260928 06:39:25.489454 25377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.2:41093 every 8 connection(s)
I20260928 06:39:25.490413 25249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-1/data/info.pb
I20260928 06:39:25.493232 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 25249
I20260928 06:39:25.493398 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-1/wal/instance
I20260928 06:39:25.493763 5208 external_mini_cluster.cc:1428] Setting key be326e66aa1a4f6e301d6b5d5649067e
I20260928 06:39:25.495903 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.22.3:0
--local_ip_for_outbound_sockets=127.5.22.3
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:41151
--builtin_ntp_servers=127.5.22.20:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:39:25.500506 25094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.492341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:54809 (local address 127.5.22.62:41151)
0928 06:39:25.492508 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:25.492513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:25.493774 (+ 1261us) server_negotiation.cc:407] Connection header received
0928 06:39:25.495490 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:25.495493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:25.495549 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:25.495657 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:25.496681 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.497218 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.498489 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.498712 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.498829 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.499989 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:25.499993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:25.500001 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:25.500014 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:25.500033 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:25.500042 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:25.500120 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:25.500294 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 06:39:25.500360 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 06:39:25.501480 25380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.492727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:41151 (local address 127.5.22.2:54809)
0928 06:39:25.493604 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:25.493648 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:25.495234 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:25.495844 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:25.495852 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:25.495943 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:25.496525 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.496539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.497364 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.497368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.498297 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.498306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.498870 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.498874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.499049 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.499706 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:39:25.499708 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:25.499749 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:39:25.499813 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:25.500135 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:25.500147 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:25.500155 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 06:39:25.500293 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 06:39:25.500591 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":169,"threads_started":1}
I20260928 06:39:25.502811 25378 heartbeater.cc:347] Connected to a master server at 127.5.22.62:41151
I20260928 06:39:25.503070 25378 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:25.503592 25378 heartbeater.cc:511] Master 127.5.22.62:41151 requested a full tablet report, sending...
I20260928 06:39:25.504685 25055 ts_manager.cc:194] Registered new tserver with Master: 6775b101cc9544d9ac1ff8c586f8432c (127.5.22.2:41093)
I20260928 06:39:25.505578 25055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.22.2:54809
W20260928 06:39:25.608456 25381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:25.608721 25381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:25.608759 25381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:25.612851 25381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:25.612902 25381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:25.612957 25381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.22.3
I20260928 06:39:25.618139 25381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.22.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.22.3
--webserver_port=0
--tserver_master_addrs=127.5.22.62:41151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.22.3
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:25.619257 25381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:25.620736 25381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:25.631039 25387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:25.631114 25381 server_base.cc:1034] running on GCE node
W20260928 06:39:25.631042 25386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:25.631084 25389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:25.631767 25381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:25.632468 25381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:25.633664 25381 hybrid_clock.cc:648] HybridClock initialized: now 1790577565633634 us; error 48 us; skew 500 ppm
I20260928 06:39:25.635690 25381 webserver.cc:460] Webserver started at http://127.5.22.3:38777/ using document root <none> and password file <none>
I20260928 06:39:25.636332 25381 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:25.636433 25381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:25.636663 25381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:25.638870 25381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-2/data/instance:
uuid: "2c5a45fc6c224113a3d3b0ba836022ad"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-j0qf"
server_key: "9f30ca8afbcfab144724875f819ac980"
server_key_iv: "45f55a28a9986ab63712a0e812e52b33"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.639441 25381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance:
uuid: "2c5a45fc6c224113a3d3b0ba836022ad"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-j0qf"
server_key: "9f30ca8afbcfab144724875f819ac980"
server_key_iv: "45f55a28a9986ab63712a0e812e52b33"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.643080 25381 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 06:39:25.645655 25395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:25.646950 25381 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 06:39:25.647112 25381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-2/wal
uuid: "2c5a45fc6c224113a3d3b0ba836022ad"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-j0qf"
server_key: "9f30ca8afbcfab144724875f819ac980"
server_key_iv: "45f55a28a9986ab63712a0e812e52b33"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.647244 25381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:25.670987 25381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:25.671800 25381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:25.672015 25381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:25.672660 25381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:25.673756 25381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:25.673831 25381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:25.673897 25381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:25.673944 25381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:25.683254 25381 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.3:45009
I20260928 06:39:25.683277 25509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.3:45009 every 8 connection(s)
I20260928 06:39:25.684213 25381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-2/data/info.pb
I20260928 06:39:25.692878 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 25381
I20260928 06:39:25.693024 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-2/wal/instance
I20260928 06:39:25.693401 5208 external_mini_cluster.cc:1428] Setting key b51ae0a0d1e5813e6d0ead75abb0e3aa
I20260928 06:39:25.694345 25094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.686143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.3:37007 (local address 127.5.22.62:41151)
0928 06:39:25.686301 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:25.686306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:25.687569 (+ 1263us) server_negotiation.cc:407] Connection header received
0928 06:39:25.689098 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:25.689100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:25.689150 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:25.689230 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:25.690169 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.690749 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.691884 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.692103 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.692190 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.693736 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:25.693740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:25.693742 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:25.693757 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:25.693780 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:25.693788 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:25.693900 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:25.694085 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 06:39:25.694159 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 06:39:25.695292 25512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.686612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:41151 (local address 127.5.22.3:37007)
0928 06:39:25.687398 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:25.687444 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:25.688884 (+ 1440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:25.689443 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:25.689450 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:25.689533 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:25.690020 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.690031 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.690889 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.690891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.691764 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.691773 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.692243 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.692246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.692561 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.693326 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:39:25.693328 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:25.693396 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:39:25.693493 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:25.693904 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:25.693918 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:25.693924 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:39:25.694072 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 06:39:25.694409 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":114,"threads_started":1}
I20260928 06:39:25.696450 25510 heartbeater.cc:347] Connected to a master server at 127.5.22.62:41151
I20260928 06:39:25.696700 25510 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:25.697280 25510 heartbeater.cc:511] Master 127.5.22.62:41151 requested a full tablet report, sending...
I20260928 06:39:25.698284 25055 ts_manager.cc:194] Registered new tserver with Master: 2c5a45fc6c224113a3d3b0ba836022ad (127.5.22.3:45009)
I20260928 06:39:25.698948 25055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.22.3:37007
I20260928 06:39:25.707890 5208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:39:25.713871 25094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.710004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46436 (local address 127.5.22.62:41151)
0928 06:39:25.710127 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:25.710131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:25.710278 (+ 147us) server_negotiation.cc:407] Connection header received
0928 06:39:25.710452 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:25.710455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:25.710497 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:25.710560 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:25.710950 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.711587 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.712435 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.712664 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.712767 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.713506 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:25.713509 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:25.713512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:25.713524 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:25.713542 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:25.713550 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:25.713614 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:25.713686 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 06:39:25.713752 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:39:25.716957 25055 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:39:25.720098 25055 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:39:25.734403 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.728662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:44447 (local address 127.0.0.1:49082)
0928 06:39:25.729390 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:25.729441 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:25.729562 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:25.730232 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:25.730240 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:25.730264 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:25.730488 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.730497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.731753 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.731759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.732952 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.732961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.733781 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.733785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.733902 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.733953 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 06:39:25.734097 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 06:39:25.734216 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":101,"threads_started":1}
I20260928 06:39:25.734452 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.728862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49082 (local address 127.5.22.1:44447)
0928 06:39:25.729675 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:25.729689 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:25.729718 (+ 29us) server_negotiation.cc:407] Connection header received
0928 06:39:25.729800 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:25.729806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:25.730026 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:25.730233 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:25.730651 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.731631 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.733084 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.733656 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.733770 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.734016 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:25.734129 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 06:39:25.734231 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":135,"threads_started":1}
I20260928 06:39:25.735188 25525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.729393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.2:41093 (local address 127.0.0.1:46864)
0928 06:39:25.730087 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:25.730097 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:25.730190 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:25.730963 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:25.730966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:25.730985 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:25.731076 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.731082 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.732479 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.732482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.733560 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.733566 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.734717 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.734721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.734814 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.734889 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 06:39:25.734971 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 06:39:25.735052 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":3,"thread_start_us":90,"threads_started":1}
I20260928 06:39:25.735620 25526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.729800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46864 (local address 127.5.22.2:41093)
0928 06:39:25.730484 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:25.730496 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:25.730520 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:39:25.730600 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:25.730608 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:25.730813 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:25.730969 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:25.731211 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.732364 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.733685 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.734495 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.734603 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.734792 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:25.735008 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 06:39:25.735112 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":90,"threads_started":1}
I20260928 06:39:25.736199 25523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.728954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:45009 (local address 127.0.0.1:47852)
0928 06:39:25.729806 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:25.729820 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:25.729899 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:25.730722 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:25.730725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:25.730744 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:25.730851 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.730859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.732333 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.732337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.733540 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.733551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.734811 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.734815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.735912 (+ 1097us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.735935 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 06:39:25.736012 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 06:39:25.736076 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":979,"thread_start_us":106,"threads_started":1}
I20260928 06:39:25.736790 25524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.729197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47852 (local address 127.5.22.3:45009)
0928 06:39:25.730233 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:25.730244 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:25.730268 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:39:25.730373 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:25.730379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:25.730548 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:25.730739 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:25.730994 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.732199 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.733711 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.734669 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.734864 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.735035 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:25.736532 (+ 1497us) server_negotiation.cc:299] Negotiation successful
0928 06:39:25.736627 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":919,"thread_start_us":134,"threads_started":1}
I20260928 06:39:25.738137 25179 tablet_service.cc:1467] Processing CreateTablet for tablet 62221e5845234bda8df3a1bdbb39def5 (DEFAULT_TABLE table=test-table [id=a7f8e528a88c4d2789d73f2cfdb1fa98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:25.739395 25179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62221e5845234bda8df3a1bdbb39def5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:25.739471 25312 tablet_service.cc:1467] Processing CreateTablet for tablet 62221e5845234bda8df3a1bdbb39def5 (DEFAULT_TABLE table=test-table [id=a7f8e528a88c4d2789d73f2cfdb1fa98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:25.739446 25444 tablet_service.cc:1467] Processing CreateTablet for tablet 62221e5845234bda8df3a1bdbb39def5 (DEFAULT_TABLE table=test-table [id=a7f8e528a88c4d2789d73f2cfdb1fa98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:25.740270 25312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62221e5845234bda8df3a1bdbb39def5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:25.740413 25444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62221e5845234bda8df3a1bdbb39def5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:25.747174 25527 tablet_bootstrap.cc:492] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c: Bootstrap starting.
I20260928 06:39:25.747820 25528 tablet_bootstrap.cc:492] T 62221e5845234bda8df3a1bdbb39def5 P 5154a49f198746aeb1a55d070ad57444: Bootstrap starting.
I20260928 06:39:25.747884 25529 tablet_bootstrap.cc:492] T 62221e5845234bda8df3a1bdbb39def5 P 2c5a45fc6c224113a3d3b0ba836022ad: Bootstrap starting.
I20260928 06:39:25.749364 25527 tablet_bootstrap.cc:654] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:25.750051 25529 tablet_bootstrap.cc:654] T 62221e5845234bda8df3a1bdbb39def5 P 2c5a45fc6c224113a3d3b0ba836022ad: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:25.750087 25528 tablet_bootstrap.cc:654] T 62221e5845234bda8df3a1bdbb39def5 P 5154a49f198746aeb1a55d070ad57444: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:25.750209 25527 log.cc:824] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:25.750918 25529 log.cc:824] T 62221e5845234bda8df3a1bdbb39def5 P 2c5a45fc6c224113a3d3b0ba836022ad: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:25.750918 25528 log.cc:824] T 62221e5845234bda8df3a1bdbb39def5 P 5154a49f198746aeb1a55d070ad57444: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:25.752959 25529 tablet_bootstrap.cc:492] T 62221e5845234bda8df3a1bdbb39def5 P 2c5a45fc6c224113a3d3b0ba836022ad: No bootstrap required, opened a new log
I20260928 06:39:25.753010 25528 tablet_bootstrap.cc:492] T 62221e5845234bda8df3a1bdbb39def5 P 5154a49f198746aeb1a55d070ad57444: No bootstrap required, opened a new log
I20260928 06:39:25.753014 25527 tablet_bootstrap.cc:492] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c: No bootstrap required, opened a new log
I20260928 06:39:25.753180 25529 ts_tablet_manager.cc:1397] T 62221e5845234bda8df3a1bdbb39def5 P 2c5a45fc6c224113a3d3b0ba836022ad: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 06:39:25.753181 25527 ts_tablet_manager.cc:1397] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260928 06:39:25.753193 25528 ts_tablet_manager.cc:1397] T 62221e5845234bda8df3a1bdbb39def5 P 5154a49f198746aeb1a55d070ad57444: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260928 06:39:25.756524 25529 raft_consensus.cc:348] T 62221e5845234bda8df3a1bdbb39def5 P 2c5a45fc6c224113a3d3b0ba836022ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5154a49f198746aeb1a55d070ad57444" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44447 } } peers { permanent_uuid: "2c5a45fc6c224113a3d3b0ba836022ad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 45009 } } peers { permanent_uuid: "6775b101cc9544d9ac1ff8c586f8432c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41093 } }
I20260928 06:39:25.756793 25529 raft_consensus.cc:374] T 62221e5845234bda8df3a1bdbb39def5 P 2c5a45fc6c224113a3d3b0ba836022ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:25.756856 25529 raft_consensus.cc:729] T 62221e5845234bda8df3a1bdbb39def5 P 2c5a45fc6c224113a3d3b0ba836022ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c5a45fc6c224113a3d3b0ba836022ad, State: Initialized, Role: FOLLOWER
I20260928 06:39:25.757355 25529 consensus_queue.cc:260] T 62221e5845234bda8df3a1bdbb39def5 P 2c5a45fc6c224113a3d3b0ba836022ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5154a49f198746aeb1a55d070ad57444" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44447 } } peers { permanent_uuid: "2c5a45fc6c224113a3d3b0ba836022ad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 45009 } } peers { permanent_uuid: "6775b101cc9544d9ac1ff8c586f8432c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41093 } }
I20260928 06:39:25.758131 25510 heartbeater.cc:503] Master 127.5.22.62:41151 was elected leader, sending a full tablet report...
I20260928 06:39:25.757992 25527 raft_consensus.cc:348] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5154a49f198746aeb1a55d070ad57444" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44447 } } peers { permanent_uuid: "2c5a45fc6c224113a3d3b0ba836022ad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 45009 } } peers { permanent_uuid: "6775b101cc9544d9ac1ff8c586f8432c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41093 } }
I20260928 06:39:25.757992 25528 raft_consensus.cc:348] T 62221e5845234bda8df3a1bdbb39def5 P 5154a49f198746aeb1a55d070ad57444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5154a49f198746aeb1a55d070ad57444" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44447 } } peers { permanent_uuid: "2c5a45fc6c224113a3d3b0ba836022ad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 45009 } } peers { permanent_uuid: "6775b101cc9544d9ac1ff8c586f8432c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41093 } }
I20260928 06:39:25.758252 25528 raft_consensus.cc:374] T 62221e5845234bda8df3a1bdbb39def5 P 5154a49f198746aeb1a55d070ad57444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:25.758252 25527 raft_consensus.cc:374] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:25.758337 25527 raft_consensus.cc:729] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6775b101cc9544d9ac1ff8c586f8432c, State: Initialized, Role: FOLLOWER
I20260928 06:39:25.758340 25528 raft_consensus.cc:729] T 62221e5845234bda8df3a1bdbb39def5 P 5154a49f198746aeb1a55d070ad57444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5154a49f198746aeb1a55d070ad57444, State: Initialized, Role: FOLLOWER
I20260928 06:39:25.758802 25527 consensus_queue.cc:260] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5154a49f198746aeb1a55d070ad57444" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44447 } } peers { permanent_uuid: "2c5a45fc6c224113a3d3b0ba836022ad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 45009 } } peers { permanent_uuid: "6775b101cc9544d9ac1ff8c586f8432c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41093 } }
I20260928 06:39:25.758791 25528 consensus_queue.cc:260] T 62221e5845234bda8df3a1bdbb39def5 P 5154a49f198746aeb1a55d070ad57444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5154a49f198746aeb1a55d070ad57444" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44447 } } peers { permanent_uuid: "2c5a45fc6c224113a3d3b0ba836022ad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 45009 } } peers { permanent_uuid: "6775b101cc9544d9ac1ff8c586f8432c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41093 } }
I20260928 06:39:25.759819 25528 ts_tablet_manager.cc:1428] T 62221e5845234bda8df3a1bdbb39def5 P 5154a49f198746aeb1a55d070ad57444: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260928 06:39:25.759829 25529 ts_tablet_manager.cc:1428] T 62221e5845234bda8df3a1bdbb39def5 P 2c5a45fc6c224113a3d3b0ba836022ad: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20260928 06:39:25.760197 25246 heartbeater.cc:503] Master 127.5.22.62:41151 was elected leader, sending a full tablet report...
I20260928 06:39:25.761878 25527 ts_tablet_manager.cc:1428] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260928 06:39:25.762234 25378 heartbeater.cc:503] Master 127.5.22.62:41151 was elected leader, sending a full tablet report...
W20260928 06:39:25.794538 25247 tablet.cc:2369] T 62221e5845234bda8df3a1bdbb39def5 P 5154a49f198746aeb1a55d070ad57444: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:39:25.935175 25511 tablet.cc:2369] T 62221e5845234bda8df3a1bdbb39def5 P 2c5a45fc6c224113a3d3b0ba836022ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:39:25.969484 25535 raft_consensus.cc:482] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:39:25.969763 25535 raft_consensus.cc:504] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5154a49f198746aeb1a55d070ad57444" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44447 } } peers { permanent_uuid: "2c5a45fc6c224113a3d3b0ba836022ad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 45009 } } peers { permanent_uuid: "6775b101cc9544d9ac1ff8c586f8432c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41093 } }
I20260928 06:39:25.971056 25535 leader_election.cc:288] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5154a49f198746aeb1a55d070ad57444 (127.5.22.1:44447), 2c5a45fc6c224113a3d3b0ba836022ad (127.5.22.3:45009)
I20260928 06:39:25.976297 25524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.971527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:49653 (local address 127.5.22.3:45009)
0928 06:39:25.971772 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:25.971777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:25.971795 (+ 18us) server_negotiation.cc:407] Connection header received
0928 06:39:25.971882 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:25.971885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:25.971937 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:25.972025 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:25.972451 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.973238 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.974954 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.975624 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.975721 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.975784 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:25.976086 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 06:39:25.976161 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260928 06:39:25.976297 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.971475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.22.2:40249 (local address 127.5.22.1:44447)
0928 06:39:25.971709 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:25.971714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:25.971801 (+ 87us) server_negotiation.cc:407] Connection header received
0928 06:39:25.971893 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:25.971896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:25.971946 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:25.972024 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:25.972455 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.973238 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.974949 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.975596 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.975719 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.975784 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:25.976086 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 06:39:25.976161 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260928 06:39:25.976361 25536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.971361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:44447 (local address 127.5.22.2:40249)
0928 06:39:25.971705 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:25.971719 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:25.971811 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:25.972053 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:25.972056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:25.972095 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:25.972323 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.972330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.973359 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.973362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.974818 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.974825 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.975756 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.975760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.975929 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.975970 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 06:39:25.976052 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 06:39:25.976174 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":292,"spinlock_wait_cycles":30208,"thread_start_us":94,"threads_started":1}
I20260928 06:39:25.976408 25380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.971334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.3:45009 (local address 127.5.22.2:49653)
0928 06:39:25.971543 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:25.971561 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:25.971662 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:25.972053 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:25.972056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:25.972096 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:25.972325 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.972330 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.973359 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.973362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.974805 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.974818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.975756 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.975760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.975923 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.975968 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 06:39:25.976052 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 06:39:25.976171 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":161,"spinlock_wait_cycles":7424}
I20260928 06:39:25.977044 25464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62221e5845234bda8df3a1bdbb39def5" candidate_uuid: "6775b101cc9544d9ac1ff8c586f8432c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c5a45fc6c224113a3d3b0ba836022ad" is_pre_election: true
I20260928 06:39:25.977079 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62221e5845234bda8df3a1bdbb39def5" candidate_uuid: "6775b101cc9544d9ac1ff8c586f8432c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5154a49f198746aeb1a55d070ad57444" is_pre_election: true
I20260928 06:39:25.977397 25464 raft_consensus.cc:2454] T 62221e5845234bda8df3a1bdbb39def5 P 2c5a45fc6c224113a3d3b0ba836022ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6775b101cc9544d9ac1ff8c586f8432c in term 0.
I20260928 06:39:25.977430 25199 raft_consensus.cc:2454] T 62221e5845234bda8df3a1bdbb39def5 P 5154a49f198746aeb1a55d070ad57444 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6775b101cc9544d9ac1ff8c586f8432c in term 0.
I20260928 06:39:25.977983 25267 leader_election.cc:302] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2c5a45fc6c224113a3d3b0ba836022ad, 5154a49f198746aeb1a55d070ad57444, 6775b101cc9544d9ac1ff8c586f8432c; no voters:
I20260928 06:39:25.978283 25535 raft_consensus.cc:2781] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:39:25.978416 25535 raft_consensus.cc:482] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:39:25.978453 25535 raft_consensus.cc:3037] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:25.979578 25535 raft_consensus.cc:504] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5154a49f198746aeb1a55d070ad57444" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44447 } } peers { permanent_uuid: "2c5a45fc6c224113a3d3b0ba836022ad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 45009 } } peers { permanent_uuid: "6775b101cc9544d9ac1ff8c586f8432c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41093 } }
I20260928 06:39:25.979903 25535 leader_election.cc:288] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c [CANDIDATE]: Term 1 election: Requested vote from peers 5154a49f198746aeb1a55d070ad57444 (127.5.22.1:44447), 2c5a45fc6c224113a3d3b0ba836022ad (127.5.22.3:45009)
I20260928 06:39:25.980283 25464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62221e5845234bda8df3a1bdbb39def5" candidate_uuid: "6775b101cc9544d9ac1ff8c586f8432c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c5a45fc6c224113a3d3b0ba836022ad"
I20260928 06:39:25.980279 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62221e5845234bda8df3a1bdbb39def5" candidate_uuid: "6775b101cc9544d9ac1ff8c586f8432c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5154a49f198746aeb1a55d070ad57444"
I20260928 06:39:25.980419 25199 raft_consensus.cc:3037] T 62221e5845234bda8df3a1bdbb39def5 P 5154a49f198746aeb1a55d070ad57444 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:25.980423 25464 raft_consensus.cc:3037] T 62221e5845234bda8df3a1bdbb39def5 P 2c5a45fc6c224113a3d3b0ba836022ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:25.981462 25199 raft_consensus.cc:2454] T 62221e5845234bda8df3a1bdbb39def5 P 5154a49f198746aeb1a55d070ad57444 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6775b101cc9544d9ac1ff8c586f8432c in term 1.
I20260928 06:39:25.981544 25464 raft_consensus.cc:2454] T 62221e5845234bda8df3a1bdbb39def5 P 2c5a45fc6c224113a3d3b0ba836022ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6775b101cc9544d9ac1ff8c586f8432c in term 1.
I20260928 06:39:25.981834 25267 leader_election.cc:302] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5154a49f198746aeb1a55d070ad57444, 6775b101cc9544d9ac1ff8c586f8432c; no voters:
I20260928 06:39:25.982013 25535 raft_consensus.cc:2781] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:25.982229 25535 raft_consensus.cc:686] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c [term 1 LEADER]: Becoming Leader. State: Replica: 6775b101cc9544d9ac1ff8c586f8432c, State: Running, Role: LEADER
I20260928 06:39:25.982622 25535 consensus_queue.cc:237] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5154a49f198746aeb1a55d070ad57444" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44447 } } peers { permanent_uuid: "2c5a45fc6c224113a3d3b0ba836022ad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 45009 } } peers { permanent_uuid: "6775b101cc9544d9ac1ff8c586f8432c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41093 } }
I20260928 06:39:25.985857 25055 catalog_manager.cc:5495] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6775b101cc9544d9ac1ff8c586f8432c (127.5.22.2). New cstate: current_term: 1 leader_uuid: "6775b101cc9544d9ac1ff8c586f8432c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5154a49f198746aeb1a55d070ad57444" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c5a45fc6c224113a3d3b0ba836022ad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 45009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6775b101cc9544d9ac1ff8c586f8432c" member_type: VOTER last_known_addr { host: "127.5.22.2" port: 41093 } health_report { overall_health: HEALTHY } } }
W20260928 06:39:25.991699 25379 tablet.cc:2369] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:39:26.030542 25526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:26.026520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46878 (local address 127.5.22.2:41093)
0928 06:39:26.026677 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:26.026682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:26.026699 (+ 17us) server_negotiation.cc:407] Connection header received
0928 06:39:26.026837 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:26.026839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:26.026899 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:26.027021 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:26.027296 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.027967 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.028894 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.029184 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.029295 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:26.029633 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:26.030076 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:26.030242 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:26.030301 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 06:39:26.030379 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 06:39:26.037050 25464 raft_consensus.cc:1261] T 62221e5845234bda8df3a1bdbb39def5 P 2c5a45fc6c224113a3d3b0ba836022ad [term 1 FOLLOWER]: Refusing update from remote peer 6775b101cc9544d9ac1ff8c586f8432c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:26.037055 25199 raft_consensus.cc:1261] T 62221e5845234bda8df3a1bdbb39def5 P 5154a49f198746aeb1a55d070ad57444 [term 1 FOLLOWER]: Refusing update from remote peer 6775b101cc9544d9ac1ff8c586f8432c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:26.037751 25537 consensus_queue.cc:1035] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c5a45fc6c224113a3d3b0ba836022ad" member_type: VOTER last_known_addr { host: "127.5.22.3" port: 45009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:26.037953 25535 consensus_queue.cc:1035] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5154a49f198746aeb1a55d070ad57444" member_type: VOTER last_known_addr { host: "127.5.22.1" port: 44447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:26.045423 25540 mvcc.cc:205] Tried to move back new op lower bound from 7334205710479294464 to 7334205710268100608. Current Snapshot: MvccSnapshot[applied={T|T < 7334205710479294464}]
I20260928 06:39:26.046988 25542 mvcc.cc:205] Tried to move back new op lower bound from 7334205710479294464 to 7334205710268100608. Current Snapshot: MvccSnapshot[applied={T|T < 7334205710479294464}]
I20260928 06:39:26.049248 25541 mvcc.cc:205] Tried to move back new op lower bound from 7334205710479294464 to 7334205710268100608. Current Snapshot: MvccSnapshot[applied={T|T < 7334205710479294464}]
I20260928 06:39:26.054745 25053 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:39:26.054916 25053 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:39:26.057183 25053 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3bf850441ba0454ea76d4980962e2740: Sending DeleteTablet for 3 replicas of tablet 62221e5845234bda8df3a1bdbb39def5
I20260928 06:39:26.057922 25179 tablet_service.cc:1514] Processing DeleteTablet for tablet 62221e5845234bda8df3a1bdbb39def5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:26 UTC) from {username='slave'} at 127.0.0.1:49082
I20260928 06:39:26.058238 25312 tablet_service.cc:1514] Processing DeleteTablet for tablet 62221e5845234bda8df3a1bdbb39def5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:26 UTC) from {username='slave'} at 127.0.0.1:46864
I20260928 06:39:26.058826 25444 tablet_service.cc:1514] Processing DeleteTablet for tablet 62221e5845234bda8df3a1bdbb39def5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:26 UTC) from {username='slave'} at 127.0.0.1:47852
I20260928 06:39:26.060077 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 25115
I20260928 06:39:26.060461 25550 tablet_replica.cc:330] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c: stopping tablet replica
I20260928 06:39:26.061218 25550 raft_consensus.cc:2229] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:39:26.061566 25550 raft_consensus.cc:2258] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:26.063016 25551 tablet_replica.cc:330] T 62221e5845234bda8df3a1bdbb39def5 P 2c5a45fc6c224113a3d3b0ba836022ad: stopping tablet replica
I20260928 06:39:26.063113 25550 ts_tablet_manager.cc:1905] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:39:26.063408 25551 raft_consensus.cc:2229] T 62221e5845234bda8df3a1bdbb39def5 P 2c5a45fc6c224113a3d3b0ba836022ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:39:26.063766 25551 raft_consensus.cc:2258] T 62221e5845234bda8df3a1bdbb39def5 P 2c5a45fc6c224113a3d3b0ba836022ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:26.065279 25551 ts_tablet_manager.cc:1905] T 62221e5845234bda8df3a1bdbb39def5 P 2c5a45fc6c224113a3d3b0ba836022ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:39:26.065948 25550 ts_tablet_manager.cc:1918] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:39:26.066056 25550 log.cc:1196] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-1/wal/wals/62221e5845234bda8df3a1bdbb39def5
I20260928 06:39:26.066450 25550 ts_tablet_manager.cc:1939] T 62221e5845234bda8df3a1bdbb39def5 P 6775b101cc9544d9ac1ff8c586f8432c: Deleting consensus metadata
I20260928 06:39:26.067625 25039 catalog_manager.cc:4841] TS 6775b101cc9544d9ac1ff8c586f8432c (127.5.22.2:41093): tablet 62221e5845234bda8df3a1bdbb39def5 (table test-table [id=a7f8e528a88c4d2789d73f2cfdb1fa98]) successfully deleted
I20260928 06:39:26.068313 25551 ts_tablet_manager.cc:1918] T 62221e5845234bda8df3a1bdbb39def5 P 2c5a45fc6c224113a3d3b0ba836022ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:39:26.068485 25551 log.cc:1196] T 62221e5845234bda8df3a1bdbb39def5 P 2c5a45fc6c224113a3d3b0ba836022ad: Deleting WAL directory at /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577308481727-5208-0/minicluster-data/ts-2/wal/wals/62221e5845234bda8df3a1bdbb39def5
I20260928 06:39:26.068830 25551 ts_tablet_manager.cc:1939] T 62221e5845234bda8df3a1bdbb39def5 P 2c5a45fc6c224113a3d3b0ba836022ad: Deleting consensus metadata
I20260928 06:39:26.069921 25039 catalog_manager.cc:4841] TS 2c5a45fc6c224113a3d3b0ba836022ad (127.5.22.3:45009): tablet 62221e5845234bda8df3a1bdbb39def5 (table test-table [id=a7f8e528a88c4d2789d73f2cfdb1fa98]) successfully deleted
W20260928 06:39:26.071921 25042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.22.1:44447 (error 108)
I20260928 06:39:26.072016 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 25249
I20260928 06:39:26.074489 25523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:26.074124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.1:44447 (local address 127.0.0.1:49098)
0928 06:39:26.074266 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:26.074384 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.1:44447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 06:39:26.074667 25042 catalog_manager.cc:4584] TS 5154a49f198746aeb1a55d070ad57444 (127.5.22.1:44447): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62221e5845234bda8df3a1bdbb39def5: Network error: Client connection negotiation failed: client connection to 127.5.22.1:44447: connect: Connection refused (error 111)
I20260928 06:39:26.078933 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 25381
I20260928 06:39:26.086755 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 25024
2026-09-28T06:39:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1267 ms)
[----------] 6 tests from EncryptionPolicyTest (11255 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:26 dist-test-slave-j0qf krb5kdc[25555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:26 dist-test-slave-j0qf krb5kdc[25555](info): set up 2 sockets
Sep 28 06:39:26 dist-test-slave-j0qf krb5kdc[25555](info): commencing operation
krb5kdc: starting...
W20260928 06:39:27.137079 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:39:27 dist-test-slave-j0qf krb5kdc[25555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:39:27.266280 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46699
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:27.376504 25568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:27.376775 25568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:27.376817 25568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:27.380733 25568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:27.380785 25568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:27.380805 25568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:27.380821 25568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:27.380836 25568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:27.385946 25568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46699
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:27.386921 25568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:27.387862 25568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:27.396361 25577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:27.396281 25573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:27.396591 25568 server_base.cc:1034] running on GCE node
W20260928 06:39:27.396272 25574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:27.397181 25568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:27.398368 25568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:27.399585 25568 hybrid_clock.cc:648] HybridClock initialized: now 1790577567399564 us; error 31 us; skew 500 ppm
Sep 28 06:39:27 dist-test-slave-j0qf krb5kdc[25555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:27.402549 25568 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:39:27.403438 25568 webserver.cc:460] Webserver started at http://127.0.0.1:44699/ using document root <none> and password file <none>
I20260928 06:39:27.404014 25568 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:27.404112 25568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:27.404387 25568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:27.406519 25568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "6d185e9ae1d54c458998fee5156bf1c8"
format_stamp: "Formatted at 2026-09-28 06:39:27 on dist-test-slave-j0qf"
server_key: "3a2866b467ed3983037757d9078836ae"
server_key_iv: "51d95eab7b1126fb8407b84ef8ef64b4"
server_key_version: "encryptionkey@0"
I20260928 06:39:27.407111 25568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "6d185e9ae1d54c458998fee5156bf1c8"
format_stamp: "Formatted at 2026-09-28 06:39:27 on dist-test-slave-j0qf"
server_key: "3a2866b467ed3983037757d9078836ae"
server_key_iv: "51d95eab7b1126fb8407b84ef8ef64b4"
server_key_version: "encryptionkey@0"
I20260928 06:39:27.410887 25568 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:39:27.413460 25584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:27.414816 25568 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:39:27.414960 25568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "6d185e9ae1d54c458998fee5156bf1c8"
format_stamp: "Formatted at 2026-09-28 06:39:27 on dist-test-slave-j0qf"
server_key: "3a2866b467ed3983037757d9078836ae"
server_key_iv: "51d95eab7b1126fb8407b84ef8ef64b4"
server_key_version: "encryptionkey@0"
I20260928 06:39:27.415083 25568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:27.432587 25568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:27.435747 25568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:27.435976 25568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:27.443475 25568 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46699
I20260928 06:39:27.443500 25636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46699 every 8 connection(s)
I20260928 06:39:27.444449 25568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:27.447402 25637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:27.452780 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 25568
I20260928 06:39:27.452970 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:27.453023 25637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d185e9ae1d54c458998fee5156bf1c8: Bootstrap starting.
I20260928 06:39:27.453329 5208 external_mini_cluster.cc:1428] Setting key 10024c9e4dc713a9295d7df32da21c84
I20260928 06:39:27.455775 25637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d185e9ae1d54c458998fee5156bf1c8: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:27.456521 25637 log.cc:824] T 00000000000000000000000000000000 P 6d185e9ae1d54c458998fee5156bf1c8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:27.458473 25637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d185e9ae1d54c458998fee5156bf1c8: No bootstrap required, opened a new log
I20260928 06:39:27.462010 25637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d185e9ae1d54c458998fee5156bf1c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d185e9ae1d54c458998fee5156bf1c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46699 } }
I20260928 06:39:27.462262 25637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d185e9ae1d54c458998fee5156bf1c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:27.462371 25637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d185e9ae1d54c458998fee5156bf1c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d185e9ae1d54c458998fee5156bf1c8, State: Initialized, Role: FOLLOWER
Sep 28 06:39:27 dist-test-slave-j0qf krb5kdc[25555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:39:27.462903 25637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d185e9ae1d54c458998fee5156bf1c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d185e9ae1d54c458998fee5156bf1c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46699 } }
I20260928 06:39:27.463063 25637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d185e9ae1d54c458998fee5156bf1c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:27.463145 25637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d185e9ae1d54c458998fee5156bf1c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:27.463253 25637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d185e9ae1d54c458998fee5156bf1c8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:27.464336 25637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d185e9ae1d54c458998fee5156bf1c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d185e9ae1d54c458998fee5156bf1c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46699 } }
I20260928 06:39:27.464727 25637 leader_election.cc:302] T 00000000000000000000000000000000 P 6d185e9ae1d54c458998fee5156bf1c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d185e9ae1d54c458998fee5156bf1c8; no voters:
I20260928 06:39:27.465085 25637 leader_election.cc:288] T 00000000000000000000000000000000 P 6d185e9ae1d54c458998fee5156bf1c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:27.465196 25642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d185e9ae1d54c458998fee5156bf1c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:27.465445 25642 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d185e9ae1d54c458998fee5156bf1c8 [term 1 LEADER]: Becoming Leader. State: Replica: 6d185e9ae1d54c458998fee5156bf1c8, State: Running, Role: LEADER
I20260928 06:39:27.465867 25642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d185e9ae1d54c458998fee5156bf1c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d185e9ae1d54c458998fee5156bf1c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46699 } }
I20260928 06:39:27.466363 25637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d185e9ae1d54c458998fee5156bf1c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:27.467741 25644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d185e9ae1d54c458998fee5156bf1c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d185e9ae1d54c458998fee5156bf1c8. Latest consensus state: current_term: 1 leader_uuid: "6d185e9ae1d54c458998fee5156bf1c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d185e9ae1d54c458998fee5156bf1c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46699 } } }
I20260928 06:39:27.467893 25644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d185e9ae1d54c458998fee5156bf1c8 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:27.468410 25643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d185e9ae1d54c458998fee5156bf1c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d185e9ae1d54c458998fee5156bf1c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d185e9ae1d54c458998fee5156bf1c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46699 } } }
I20260928 06:39:27.468973 25643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d185e9ae1d54c458998fee5156bf1c8 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:27.470527 25640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:27.454803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41889 (local address 127.0.0.1:46699)
0928 06:39:27.455316 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:27.455334 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:27.455373 (+ 39us) server_negotiation.cc:407] Connection header received
0928 06:39:27.456140 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:27.456160 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:27.456557 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:27.456957 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:27.457839 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:27.459078 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:27.460019 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:27.460550 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:27.460741 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:27.463717 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:27.463722 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:27.463736 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:27.463764 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:27.466086 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:27.466721 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:27.466728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:27.466734 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:27.466886 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:27.467378 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:27.467384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:27.467387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:27.467928 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:27.468186 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:27.469124 (+ 938us) server_negotiation.cc:299] Negotiation successful
0928 06:39:27.469381 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":139,"threads_started":1}
W20260928 06:39:27.471931 25657 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6d185e9ae1d54c458998fee5156bf1c8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:39:27.472008 25657 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:39:27.472077 25653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:27.472981 25653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:27.473977 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:27.479223 25653 catalog_manager.cc:1347] Generated new cluster ID: 90b96b0aa34e4185b67e7ddca3a21638
I20260928 06:39:27.479288 25653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:27.485258 25640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:27.476263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39355 (local address 127.0.0.1:46699)
0928 06:39:27.476379 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:27.476383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:27.476607 (+ 224us) server_negotiation.cc:407] Connection header received
0928 06:39:27.476778 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:27.476781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:27.476824 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:27.476919 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:27.477859 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:27.478488 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:27.479551 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:27.479909 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:27.480025 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:27.481600 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:27.481604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:27.481607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:27.481634 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:27.483417 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:27.483940 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:27.483944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:27.483946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:27.484035 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:27.484366 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:27.484369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:27.484371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:27.484579 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:27.484720 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:27.484996 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 06:39:27.485098 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:39:27.487566 25653 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:27.488445 25653 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:27.500312 25653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d185e9ae1d54c458998fee5156bf1c8: Generated new TSK 0
I20260928 06:39:27.501102 25653 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:27.510849 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 25568
W20260928 06:39:27.516877 25553 connection.cc:535] client connection to 127.0.0.1:46699 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46699): Connection reset by peer (error 104)
2026-09-28T06:39:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:27 dist-test-slave-j0qf krb5kdc[25668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:27 dist-test-slave-j0qf krb5kdc[25668](info): set up 2 sockets
Sep 28 06:39:27 dist-test-slave-j0qf krb5kdc[25668](info): commencing operation
krb5kdc: starting...
W20260928 06:39:28.560963 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:39:28 dist-test-slave-j0qf krb5kdc[25668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:39:28.685304 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:28.798735 25681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:28.799053 25681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:28.799125 25681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:28.803165 25681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:28.803267 25681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:28.803301 25681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:28.803339 25681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:28.803368 25681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:28.808789 25681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42305
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:28.809922 25681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:28.811079 25681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:28.819991 25686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:28.820058 25687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:28.820000 25689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:28.820164 25681 server_base.cc:1034] running on GCE node
I20260928 06:39:28.821015 25681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:28.822196 25681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:28.823410 25681 hybrid_clock.cc:648] HybridClock initialized: now 1790577568823376 us; error 49 us; skew 500 ppm
Sep 28 06:39:28 dist-test-slave-j0qf krb5kdc[25668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:28.826458 25681 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:39:28.827447 25681 webserver.cc:460] Webserver started at http://127.0.0.1:34469/ using document root <none> and password file <none>
I20260928 06:39:28.828038 25681 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:28.828135 25681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:28.828361 25681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:28.830530 25681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "302db142ac2b4686ae054033a1e556d6"
format_stamp: "Formatted at 2026-09-28 06:39:28 on dist-test-slave-j0qf"
server_key: "cebbe3505296ef38c681ced3e4b0b145"
server_key_iv: "cbaea2c94b5809ac8aae9a40a2f21239"
server_key_version: "encryptionkey@0"
I20260928 06:39:28.831115 25681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "302db142ac2b4686ae054033a1e556d6"
format_stamp: "Formatted at 2026-09-28 06:39:28 on dist-test-slave-j0qf"
server_key: "cebbe3505296ef38c681ced3e4b0b145"
server_key_iv: "cbaea2c94b5809ac8aae9a40a2f21239"
server_key_version: "encryptionkey@0"
I20260928 06:39:28.835062 25681 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 06:39:28.837507 25696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:28.838806 25681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:39:28.838944 25681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "302db142ac2b4686ae054033a1e556d6"
format_stamp: "Formatted at 2026-09-28 06:39:28 on dist-test-slave-j0qf"
server_key: "cebbe3505296ef38c681ced3e4b0b145"
server_key_iv: "cbaea2c94b5809ac8aae9a40a2f21239"
server_key_version: "encryptionkey@0"
I20260928 06:39:28.839063 25681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:28.857874 25681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:28.861192 25681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:28.861420 25681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:28.869170 25681 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42305
I20260928 06:39:28.869184 25749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42305 every 8 connection(s)
I20260928 06:39:28.870167 25681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:28.872315 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 25681
I20260928 06:39:28.872473 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:28.872802 5208 external_mini_cluster.cc:1428] Setting key e491c97a78bcc512ecabe4f9ce9a9b6f
I20260928 06:39:28.873304 25750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:28.879540 25750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 302db142ac2b4686ae054033a1e556d6: Bootstrap starting.
I20260928 06:39:28.881875 25750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 302db142ac2b4686ae054033a1e556d6: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:28.882468 25753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:28.874722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51689 (local address 127.0.0.1:42305)
0928 06:39:28.875180 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:28.875195 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:28.875239 (+ 44us) server_negotiation.cc:407] Connection header received
0928 06:39:28.876080 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:28.876099 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:28.876522 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:28.876862 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:28.877269 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.878228 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.879146 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.879676 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.879854 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:28.880752 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:28.880757 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:28.880769 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:28.881139 (+ 370us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:28.881189 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:28.881220 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:28.881361 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:28.881543 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 06:39:28.881680 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":143,"threads_started":1}
I20260928 06:39:28.882742 25750 log.cc:824] T 00000000000000000000000000000000 P 302db142ac2b4686ae054033a1e556d6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:28.884719 25750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 302db142ac2b4686ae054033a1e556d6: No bootstrap required, opened a new log
I20260928 06:39:28.885311 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:28.888857 25750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 302db142ac2b4686ae054033a1e556d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302db142ac2b4686ae054033a1e556d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42305 } }
I20260928 06:39:28.889135 25750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 302db142ac2b4686ae054033a1e556d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:28.889212 25750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 302db142ac2b4686ae054033a1e556d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 302db142ac2b4686ae054033a1e556d6, State: Initialized, Role: FOLLOWER
I20260928 06:39:28.889648 25750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 302db142ac2b4686ae054033a1e556d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302db142ac2b4686ae054033a1e556d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42305 } }
I20260928 06:39:28.889816 25750 raft_consensus.cc:388] T 00000000000000000000000000000000 P 302db142ac2b4686ae054033a1e556d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:28.889920 25750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 302db142ac2b4686ae054033a1e556d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:28.890048 25750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 302db142ac2b4686ae054033a1e556d6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:28.891240 25750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 302db142ac2b4686ae054033a1e556d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302db142ac2b4686ae054033a1e556d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42305 } }
I20260928 06:39:28.891863 25753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:28.887310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38379 (local address 127.0.0.1:42305)
0928 06:39:28.887423 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:28.887427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:28.887614 (+ 187us) server_negotiation.cc:407] Connection header received
0928 06:39:28.887726 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:28.887728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:28.887760 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:28.887822 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:28.888134 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.888671 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.889857 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.890304 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.890452 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:28.891303 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:28.891306 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:28.891308 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:28.891319 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:28.891373 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:28.891389 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:28.891508 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:28.891689 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 06:39:28.891725 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:39:28.891805 25750 leader_election.cc:302] T 00000000000000000000000000000000 P 302db142ac2b4686ae054033a1e556d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 302db142ac2b4686ae054033a1e556d6; no voters:
I20260928 06:39:28.892274 25750 leader_election.cc:288] T 00000000000000000000000000000000 P 302db142ac2b4686ae054033a1e556d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:28.892380 25760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 302db142ac2b4686ae054033a1e556d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:28.892645 25760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 302db142ac2b4686ae054033a1e556d6 [term 1 LEADER]: Becoming Leader. State: Replica: 302db142ac2b4686ae054033a1e556d6, State: Running, Role: LEADER
I20260928 06:39:28.893030 25760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 302db142ac2b4686ae054033a1e556d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302db142ac2b4686ae054033a1e556d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42305 } }
I20260928 06:39:28.893599 25750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 302db142ac2b4686ae054033a1e556d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:28.894389 25760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 302db142ac2b4686ae054033a1e556d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 302db142ac2b4686ae054033a1e556d6. Latest consensus state: current_term: 1 leader_uuid: "302db142ac2b4686ae054033a1e556d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302db142ac2b4686ae054033a1e556d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42305 } } }
I20260928 06:39:28.894624 25760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 302db142ac2b4686ae054033a1e556d6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:28.894797 25761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 302db142ac2b4686ae054033a1e556d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "302db142ac2b4686ae054033a1e556d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302db142ac2b4686ae054033a1e556d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42305 } } }
I20260928 06:39:28.894950 25761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 302db142ac2b4686ae054033a1e556d6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:28.895083 25769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:28.898190 25769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:28.904578 25769 catalog_manager.cc:1347] Generated new cluster ID: 456766f6d8a943c89633720fec5dfd21
I20260928 06:39:28.904668 25769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:28.927500 25769 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:28.928488 25769 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:28.944718 25769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 302db142ac2b4686ae054033a1e556d6: Generated new TSK 0
I20260928 06:39:28.945717 25769 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:28.949536 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 25681
W20260928 06:39:28.955000 25666 connection.cc:535] client connection to 127.0.0.1:42305 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42305): Connection reset by peer (error 104)
2026-09-28T06:39:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:29 dist-test-slave-j0qf krb5kdc[25781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:29 dist-test-slave-j0qf krb5kdc[25781](info): set up 2 sockets
Sep 28 06:39:29 dist-test-slave-j0qf krb5kdc[25781](info): commencing operation
krb5kdc: starting...
W20260928 06:39:30.008980 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:39:30 dist-test-slave-j0qf krb5kdc[25781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:39:30.138957 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38353
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:30.249665 25794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:30.249985 25794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:30.250067 25794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:30.254122 25794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:30.254217 25794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:30.254258 25794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:30.254313 25794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:30.254379 25794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:30.259611 25794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38353
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:30.260643 25794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:30.261690 25794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:30.270162 25799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:30.270148 25802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:30.270129 25800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:30.270610 25794 server_base.cc:1034] running on GCE node
I20260928 06:39:30.271152 25794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:30.272302 25794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:30.273501 25794 hybrid_clock.cc:648] HybridClock initialized: now 1790577570273472 us; error 42 us; skew 500 ppm
Sep 28 06:39:30 dist-test-slave-j0qf krb5kdc[25781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:30.281235 25794 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:39:30.282229 25794 webserver.cc:460] Webserver started at http://127.0.0.1:40069/ using document root <none> and password file <none>
I20260928 06:39:30.282862 25794 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:30.282963 25794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:30.283202 25794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:30.285400 25794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "fd7c8c27c2b342ff949241c942097575"
format_stamp: "Formatted at 2026-09-28 06:39:30 on dist-test-slave-j0qf"
server_key: "d4f32591d8ff87d1e4dc40e6042470d8"
server_key_iv: "9c62d5e3a584faeb661d460ef662ec48"
server_key_version: "encryptionkey@0"
I20260928 06:39:30.285974 25794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "fd7c8c27c2b342ff949241c942097575"
format_stamp: "Formatted at 2026-09-28 06:39:30 on dist-test-slave-j0qf"
server_key: "d4f32591d8ff87d1e4dc40e6042470d8"
server_key_iv: "9c62d5e3a584faeb661d460ef662ec48"
server_key_version: "encryptionkey@0"
I20260928 06:39:30.289894 25794 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 06:39:30.292608 25809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:30.293851 25794 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:39:30.294008 25794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "fd7c8c27c2b342ff949241c942097575"
format_stamp: "Formatted at 2026-09-28 06:39:30 on dist-test-slave-j0qf"
server_key: "d4f32591d8ff87d1e4dc40e6042470d8"
server_key_iv: "9c62d5e3a584faeb661d460ef662ec48"
server_key_version: "encryptionkey@0"
I20260928 06:39:30.294137 25794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:30.298987 25794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:30.302277 25794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:30.302541 25794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:30.310017 25794 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38353
I20260928 06:39:30.310041 25861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38353 every 8 connection(s)
I20260928 06:39:30.311057 25794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:30.313894 25862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:30.315657 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 25794
I20260928 06:39:30.315817 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:30.316102 5208 external_mini_cluster.cc:1428] Setting key fed90fbbf2d5adfbcef66acc2e0e5af2
I20260928 06:39:30.320499 25862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd7c8c27c2b342ff949241c942097575: Bootstrap starting.
I20260928 06:39:30.321554 25865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:30.317598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49449 (local address 127.0.0.1:38353)
0928 06:39:30.318082 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:30.318097 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:30.318133 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:39:30.318860 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:30.318879 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:30.319172 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:30.319530 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:30.320366 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:30.320370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:30.320382 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:30.320393 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:30.320447 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:30.320467 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:30.320621 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:30.320704 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 06:39:30.320863 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":157,"threads_started":1}
I20260928 06:39:30.323053 25862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd7c8c27c2b342ff949241c942097575: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:30.323961 25862 log.cc:824] T 00000000000000000000000000000000 P fd7c8c27c2b342ff949241c942097575: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:30.324493 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:30.326042 25862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd7c8c27c2b342ff949241c942097575: No bootstrap required, opened a new log
I20260928 06:39:30.328604 25865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:30.326745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36611 (local address 127.0.0.1:38353)
0928 06:39:30.326881 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:30.326885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:30.327034 (+ 149us) server_negotiation.cc:407] Connection header received
0928 06:39:30.327198 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:30.327204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:30.327241 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:30.327336 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:30.328117 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:30.328121 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:30.328123 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:30.328135 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:30.328162 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:30.328179 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:30.328263 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:30.328442 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 06:39:30.328483 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 06:39:30.330569 25862 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd7c8c27c2b342ff949241c942097575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd7c8c27c2b342ff949241c942097575" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38353 } }
I20260928 06:39:30.330826 25862 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd7c8c27c2b342ff949241c942097575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:30.330911 25862 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd7c8c27c2b342ff949241c942097575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd7c8c27c2b342ff949241c942097575, State: Initialized, Role: FOLLOWER
I20260928 06:39:30.331328 25862 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd7c8c27c2b342ff949241c942097575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd7c8c27c2b342ff949241c942097575" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38353 } }
I20260928 06:39:30.331492 25862 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd7c8c27c2b342ff949241c942097575 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:30.331584 25862 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd7c8c27c2b342ff949241c942097575 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:30.331709 25862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd7c8c27c2b342ff949241c942097575 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:30.332731 25862 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd7c8c27c2b342ff949241c942097575 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd7c8c27c2b342ff949241c942097575" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38353 } }
I20260928 06:39:30.333079 25862 leader_election.cc:302] T 00000000000000000000000000000000 P fd7c8c27c2b342ff949241c942097575 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd7c8c27c2b342ff949241c942097575; no voters:
I20260928 06:39:30.333400 25862 leader_election.cc:288] T 00000000000000000000000000000000 P fd7c8c27c2b342ff949241c942097575 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:30.333503 25872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd7c8c27c2b342ff949241c942097575 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:30.333714 25872 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd7c8c27c2b342ff949241c942097575 [term 1 LEADER]: Becoming Leader. State: Replica: fd7c8c27c2b342ff949241c942097575, State: Running, Role: LEADER
I20260928 06:39:30.334048 25872 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd7c8c27c2b342ff949241c942097575 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd7c8c27c2b342ff949241c942097575" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38353 } }
I20260928 06:39:30.334713 25862 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd7c8c27c2b342ff949241c942097575 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:30.335954 25873 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd7c8c27c2b342ff949241c942097575 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd7c8c27c2b342ff949241c942097575" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd7c8c27c2b342ff949241c942097575" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38353 } } }
I20260928 06:39:30.335955 25874 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd7c8c27c2b342ff949241c942097575 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd7c8c27c2b342ff949241c942097575. Latest consensus state: current_term: 1 leader_uuid: "fd7c8c27c2b342ff949241c942097575" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd7c8c27c2b342ff949241c942097575" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38353 } } }
I20260928 06:39:30.336105 25874 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd7c8c27c2b342ff949241c942097575 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:30.336105 25873 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd7c8c27c2b342ff949241c942097575 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:30.337075 25881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:30.340093 25881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:30.345754 25881 catalog_manager.cc:1347] Generated new cluster ID: 6de59d8878b04c3c9047691fe6db4696
I20260928 06:39:30.345825 25881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:30.373888 25881 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:30.374246 25881 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:30.384804 25881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd7c8c27c2b342ff949241c942097575: Generated new TSK 0
I20260928 06:39:30.385550 25881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:30.388588 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 25794
2026-09-28T06:39:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1438 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:30 dist-test-slave-j0qf krb5kdc[25893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:30 dist-test-slave-j0qf krb5kdc[25893](info): set up 2 sockets
Sep 28 06:39:30 dist-test-slave-j0qf krb5kdc[25893](info): commencing operation
krb5kdc: starting...
W20260928 06:39:31.424912 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:39:31 dist-test-slave-j0qf krb5kdc[25893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:39:31.553877 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33933
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:31.663784 25907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:31.664063 25907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:31.664106 25907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:31.668306 25907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:31.668380 25907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:31.668401 25907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:31.668419 25907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:31.668434 25907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:31.673879 25907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:31.675055 25907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:31.676101 25907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:31.684803 25913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:31.684823 25915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:31.684971 25907 server_base.cc:1034] running on GCE node
W20260928 06:39:31.684832 25912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:31.685684 25907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:31.686866 25907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:31.688055 25907 hybrid_clock.cc:648] HybridClock initialized: now 1790577571688042 us; error 28 us; skew 500 ppm
Sep 28 06:39:31 dist-test-slave-j0qf krb5kdc[25893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:31.690855 25907 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:39:31.691730 25907 webserver.cc:460] Webserver started at http://127.0.0.1:38181/ using document root <none> and password file <none>
I20260928 06:39:31.692303 25907 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:31.692399 25907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:31.692642 25907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:31.694737 25907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "cbe47c29438f4cab8f9c7304cd16bea5"
format_stamp: "Formatted at 2026-09-28 06:39:31 on dist-test-slave-j0qf"
server_key: "2323e8bba95555946ed03a960a0250cc"
server_key_iv: "b335707f8003fd1846d1781e655c6ef9"
server_key_version: "encryptionkey@0"
I20260928 06:39:31.695302 25907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "cbe47c29438f4cab8f9c7304cd16bea5"
format_stamp: "Formatted at 2026-09-28 06:39:31 on dist-test-slave-j0qf"
server_key: "2323e8bba95555946ed03a960a0250cc"
server_key_iv: "b335707f8003fd1846d1781e655c6ef9"
server_key_version: "encryptionkey@0"
I20260928 06:39:31.699126 25907 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 06:39:31.701538 25923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:31.702812 25907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 06:39:31.702962 25907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "cbe47c29438f4cab8f9c7304cd16bea5"
format_stamp: "Formatted at 2026-09-28 06:39:31 on dist-test-slave-j0qf"
server_key: "2323e8bba95555946ed03a960a0250cc"
server_key_iv: "b335707f8003fd1846d1781e655c6ef9"
server_key_version: "encryptionkey@0"
I20260928 06:39:31.703078 25907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:31.723678 25907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:31.727100 25907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:31.727316 25907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:31.734761 25907 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33933
I20260928 06:39:31.734786 25975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33933 every 8 connection(s)
I20260928 06:39:31.735713 25907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:31.738641 25976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:31.740351 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 25907
I20260928 06:39:31.740494 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:31.740775 5208 external_mini_cluster.cc:1428] Setting key 0909c291837f7fbe44fa10bc20287ae6
I20260928 06:39:31.745639 25976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbe47c29438f4cab8f9c7304cd16bea5: Bootstrap starting.
I20260928 06:39:31.748363 25976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbe47c29438f4cab8f9c7304cd16bea5: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:31.749267 25976 log.cc:824] T 00000000000000000000000000000000 P cbe47c29438f4cab8f9c7304cd16bea5: Log is configured to *not* fsync() on all Append() calls
Sep 28 06:39:31 dist-test-slave-j0qf krb5kdc[25893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:39:31.751829 25976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbe47c29438f4cab8f9c7304cd16bea5: No bootstrap required, opened a new log
I20260928 06:39:31.756016 25979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:31.742233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35565 (local address 127.0.0.1:33933)
0928 06:39:31.742729 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:31.742750 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:31.742794 (+ 44us) server_negotiation.cc:407] Connection header received
0928 06:39:31.743697 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:31.743721 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:31.744164 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:31.744538 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:31.745209 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.746674 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.747538 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.748238 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.748414 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:31.751405 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:31.751410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:31.751423 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:31.751448 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:31.753130 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:31.753626 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:31.753630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:31.753635 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:31.753748 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:31.754078 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:31.754081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:31.754082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:31.754524 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:31.754677 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:31.754975 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 06:39:31.755264 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":110,"threads_started":1}
I20260928 06:39:31.756156 25976 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbe47c29438f4cab8f9c7304cd16bea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe47c29438f4cab8f9c7304cd16bea5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33933 } }
I20260928 06:39:31.756448 25976 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbe47c29438f4cab8f9c7304cd16bea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:31.756548 25976 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbe47c29438f4cab8f9c7304cd16bea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbe47c29438f4cab8f9c7304cd16bea5, State: Initialized, Role: FOLLOWER
I20260928 06:39:31.756994 25976 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbe47c29438f4cab8f9c7304cd16bea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe47c29438f4cab8f9c7304cd16bea5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33933 } }
I20260928 06:39:31.757169 25976 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbe47c29438f4cab8f9c7304cd16bea5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:31.757229 25976 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbe47c29438f4cab8f9c7304cd16bea5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:31.757323 25976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbe47c29438f4cab8f9c7304cd16bea5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:31.758566 25976 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbe47c29438f4cab8f9c7304cd16bea5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe47c29438f4cab8f9c7304cd16bea5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33933 } }
I20260928 06:39:31.758940 25976 leader_election.cc:302] T 00000000000000000000000000000000 P cbe47c29438f4cab8f9c7304cd16bea5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbe47c29438f4cab8f9c7304cd16bea5; no voters:
I20260928 06:39:31.759323 25976 leader_election.cc:288] T 00000000000000000000000000000000 P cbe47c29438f4cab8f9c7304cd16bea5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:31.759352 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:31.759409 25981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbe47c29438f4cab8f9c7304cd16bea5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:31.759663 25981 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbe47c29438f4cab8f9c7304cd16bea5 [term 1 LEADER]: Becoming Leader. State: Replica: cbe47c29438f4cab8f9c7304cd16bea5, State: Running, Role: LEADER
I20260928 06:39:31.760044 25981 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbe47c29438f4cab8f9c7304cd16bea5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe47c29438f4cab8f9c7304cd16bea5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33933 } }
I20260928 06:39:31.760761 25976 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbe47c29438f4cab8f9c7304cd16bea5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:31.761345 25981 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbe47c29438f4cab8f9c7304cd16bea5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbe47c29438f4cab8f9c7304cd16bea5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe47c29438f4cab8f9c7304cd16bea5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33933 } } }
I20260928 06:39:31.761526 25981 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbe47c29438f4cab8f9c7304cd16bea5 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:31.761615 25982 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbe47c29438f4cab8f9c7304cd16bea5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbe47c29438f4cab8f9c7304cd16bea5. Latest consensus state: current_term: 1 leader_uuid: "cbe47c29438f4cab8f9c7304cd16bea5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe47c29438f4cab8f9c7304cd16bea5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33933 } } }
I20260928 06:39:31.761752 25982 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbe47c29438f4cab8f9c7304cd16bea5 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:31.762080 25991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:31.765498 25991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:31.771869 25991 catalog_manager.cc:1347] Generated new cluster ID: 38618589a9b443ec97c91069137a964e
I20260928 06:39:31.772004 25991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:31.772969 25979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:31.763174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50215 (local address 127.0.0.1:33933)
0928 06:39:31.763290 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:31.763295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:31.763661 (+ 366us) server_negotiation.cc:407] Connection header received
0928 06:39:31.763803 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:31.763806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:31.763876 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:31.763999 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:31.764847 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.765784 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.766824 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.767142 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.767251 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:31.768822 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:31.768827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:31.768830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:31.768861 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:31.770715 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:31.771247 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:31.771251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:31.771253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:31.771351 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:31.771678 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:31.771681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:31.771683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:31.771900 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:31.772603 (+ 703us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:31.772679 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 06:39:31.772797 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:39:31.786859 25991 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:31.788163 25991 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:31.795085 25991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbe47c29438f4cab8f9c7304cd16bea5: Generated new TSK 0
I20260928 06:39:31.795949 25991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:31.804332 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 25907
2026-09-28T06:39:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:31 dist-test-slave-j0qf krb5kdc[26008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:31 dist-test-slave-j0qf krb5kdc[26008](info): set up 2 sockets
Sep 28 06:39:31 dist-test-slave-j0qf krb5kdc[26008](info): commencing operation
krb5kdc: starting...
W20260928 06:39:32.861025 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:39:32 dist-test-slave-j0qf krb5kdc[26008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:39:32.993345 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44263
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:33.105166 26021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:33.105496 26021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:33.105579 26021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:33.109743 26021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:33.109839 26021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:33.109877 26021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:33.109935 26021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:33.109968 26021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:33.115447 26021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:33.116606 26021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:33.117697 26021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:33.126537 26029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:33.126541 26026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:33.126780 26021 server_base.cc:1034] running on GCE node
W20260928 06:39:33.126626 26027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:33.127400 26021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:33.128602 26021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:33.129868 26021 hybrid_clock.cc:648] HybridClock initialized: now 1790577573129854 us; error 34 us; skew 500 ppm
Sep 28 06:39:33 dist-test-slave-j0qf krb5kdc[26008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:33.132827 26021 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:39:33.133749 26021 webserver.cc:460] Webserver started at http://127.0.0.1:38175/ using document root <none> and password file <none>
I20260928 06:39:33.134428 26021 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:33.134524 26021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:33.134742 26021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:33.136824 26021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "cf9b08fdb2d44ee3b06c2831b6d60354"
format_stamp: "Formatted at 2026-09-28 06:39:33 on dist-test-slave-j0qf"
server_key: "db3c05f7c569d5eaaeb746008a8111dc"
server_key_iv: "84dceebb2be41e13a0ac3bb7b3bb5786"
server_key_version: "encryptionkey@0"
I20260928 06:39:33.137416 26021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "cf9b08fdb2d44ee3b06c2831b6d60354"
format_stamp: "Formatted at 2026-09-28 06:39:33 on dist-test-slave-j0qf"
server_key: "db3c05f7c569d5eaaeb746008a8111dc"
server_key_iv: "84dceebb2be41e13a0ac3bb7b3bb5786"
server_key_version: "encryptionkey@0"
I20260928 06:39:33.141302 26021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 06:39:33.143695 26036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:33.144904 26021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:39:33.145056 26021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "cf9b08fdb2d44ee3b06c2831b6d60354"
format_stamp: "Formatted at 2026-09-28 06:39:33 on dist-test-slave-j0qf"
server_key: "db3c05f7c569d5eaaeb746008a8111dc"
server_key_iv: "84dceebb2be41e13a0ac3bb7b3bb5786"
server_key_version: "encryptionkey@0"
I20260928 06:39:33.145171 26021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:33.173612 26021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:33.177062 26021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:33.177276 26021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:33.184541 26021 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44263
I20260928 06:39:33.184572 26089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44263 every 8 connection(s)
I20260928 06:39:33.185465 26021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:33.188203 26090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:33.190495 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 26021
I20260928 06:39:33.190624 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:33.190883 5208 external_mini_cluster.cc:1428] Setting key f1162fddef43ffc0849d6c2aa0ab3bf6
I20260928 06:39:33.194043 26090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf9b08fdb2d44ee3b06c2831b6d60354: Bootstrap starting.
I20260928 06:39:33.196296 26090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf9b08fdb2d44ee3b06c2831b6d60354: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:33.197027 26090 log.cc:824] T 00000000000000000000000000000000 P cf9b08fdb2d44ee3b06c2831b6d60354: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:33.198979 26090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf9b08fdb2d44ee3b06c2831b6d60354: No bootstrap required, opened a new log
I20260928 06:39:33.199769 26093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:33.192275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38985 (local address 127.0.0.1:44263)
0928 06:39:33.192730 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:33.192745 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:33.192781 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:39:33.193432 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:33.193450 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:33.193799 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:33.194139 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:33.194571 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:33.195538 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:33.196461 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:33.196953 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:33.197143 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:33.197994 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:33.198000 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:33.198012 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:33.198471 (+ 459us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:33.198527 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:33.198552 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:33.198678 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:33.198795 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 06:39:33.199006 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":136,"threads_started":1}
I20260928 06:39:33.202591 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:33.202693 26090 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf9b08fdb2d44ee3b06c2831b6d60354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf9b08fdb2d44ee3b06c2831b6d60354" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44263 } }
I20260928 06:39:33.202945 26090 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf9b08fdb2d44ee3b06c2831b6d60354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:33.203028 26090 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf9b08fdb2d44ee3b06c2831b6d60354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf9b08fdb2d44ee3b06c2831b6d60354, State: Initialized, Role: FOLLOWER
I20260928 06:39:33.203490 26090 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf9b08fdb2d44ee3b06c2831b6d60354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf9b08fdb2d44ee3b06c2831b6d60354" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44263 } }
I20260928 06:39:33.203610 26090 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf9b08fdb2d44ee3b06c2831b6d60354 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:33.203688 26090 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf9b08fdb2d44ee3b06c2831b6d60354 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:33.203783 26090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf9b08fdb2d44ee3b06c2831b6d60354 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:33.204932 26090 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf9b08fdb2d44ee3b06c2831b6d60354 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf9b08fdb2d44ee3b06c2831b6d60354" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44263 } }
I20260928 06:39:33.205294 26090 leader_election.cc:302] T 00000000000000000000000000000000 P cf9b08fdb2d44ee3b06c2831b6d60354 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf9b08fdb2d44ee3b06c2831b6d60354; no voters:
I20260928 06:39:33.205608 26090 leader_election.cc:288] T 00000000000000000000000000000000 P cf9b08fdb2d44ee3b06c2831b6d60354 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:33.205724 26100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf9b08fdb2d44ee3b06c2831b6d60354 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:33.205952 26100 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf9b08fdb2d44ee3b06c2831b6d60354 [term 1 LEADER]: Becoming Leader. State: Replica: cf9b08fdb2d44ee3b06c2831b6d60354, State: Running, Role: LEADER
I20260928 06:39:33.206280 26100 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf9b08fdb2d44ee3b06c2831b6d60354 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf9b08fdb2d44ee3b06c2831b6d60354" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44263 } }
I20260928 06:39:33.206861 26090 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf9b08fdb2d44ee3b06c2831b6d60354 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:33.207491 26100 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf9b08fdb2d44ee3b06c2831b6d60354 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf9b08fdb2d44ee3b06c2831b6d60354. Latest consensus state: current_term: 1 leader_uuid: "cf9b08fdb2d44ee3b06c2831b6d60354" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf9b08fdb2d44ee3b06c2831b6d60354" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44263 } } }
I20260928 06:39:33.207669 26100 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf9b08fdb2d44ee3b06c2831b6d60354 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:33.208101 26108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:33.209882 26102 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf9b08fdb2d44ee3b06c2831b6d60354 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf9b08fdb2d44ee3b06c2831b6d60354" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf9b08fdb2d44ee3b06c2831b6d60354" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44263 } } }
I20260928 06:39:33.210012 26093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:33.204801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48699 (local address 127.0.0.1:44263)
0928 06:39:33.204922 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:33.204925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:33.205199 (+ 274us) server_negotiation.cc:407] Connection header received
0928 06:39:33.205351 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:33.205354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:33.205397 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:33.205494 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:33.205864 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:33.206648 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:33.207877 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:33.208222 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:33.208302 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:33.209467 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:33.209471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:33.209474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:33.209487 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:33.209514 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:33.209527 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:33.209606 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:33.209774 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 06:39:33.209833 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:39:33.210112 26102 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf9b08fdb2d44ee3b06c2831b6d60354 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:33.211957 26108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:33.217793 26108 catalog_manager.cc:1347] Generated new cluster ID: adc32a61b22f45aea5315c160a5097c1
I20260928 06:39:33.217864 26108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:33.236200 26108 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:33.237195 26108 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:33.243561 26108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf9b08fdb2d44ee3b06c2831b6d60354: Generated new TSK 0
I20260928 06:39:33.244410 26108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:33.256820 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 26021
2026-09-28T06:39:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1452 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:33 dist-test-slave-j0qf krb5kdc[26121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:33 dist-test-slave-j0qf krb5kdc[26121](info): set up 2 sockets
Sep 28 06:39:33 dist-test-slave-j0qf krb5kdc[26121](info): commencing operation
krb5kdc: starting...
W20260928 06:39:34.301039 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:39:34 dist-test-slave-j0qf krb5kdc[26121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:39:34.433516 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34999
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:34.548162 26134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:34.548478 26134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:34.548553 26134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:34.552626 26134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:34.552724 26134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:34.552762 26134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:34.552822 26134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:34.552860 26134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:34.558194 26134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:34.559285 26134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:34.560322 26134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:34.569114 26140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:34.569125 26139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:34.569125 26142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:34.569182 26134 server_base.cc:1034] running on GCE node
I20260928 06:39:34.570012 26134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:34.571193 26134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:34.572396 26134 hybrid_clock.cc:648] HybridClock initialized: now 1790577574572373 us; error 26 us; skew 500 ppm
Sep 28 06:39:34 dist-test-slave-j0qf krb5kdc[26121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:34.575220 26134 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:39:34.576170 26134 webserver.cc:460] Webserver started at http://127.0.0.1:38151/ using document root <none> and password file <none>
I20260928 06:39:34.576743 26134 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:34.576839 26134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:34.577054 26134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:34.579195 26134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "1698231d854a4094b87635706b8245d2"
format_stamp: "Formatted at 2026-09-28 06:39:34 on dist-test-slave-j0qf"
server_key: "7728c44c1f8134e7488ad5b71bb49cba"
server_key_iv: "995c85cd6d438de33e611eeadd63d0d4"
server_key_version: "encryptionkey@0"
I20260928 06:39:34.579803 26134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "1698231d854a4094b87635706b8245d2"
format_stamp: "Formatted at 2026-09-28 06:39:34 on dist-test-slave-j0qf"
server_key: "7728c44c1f8134e7488ad5b71bb49cba"
server_key_iv: "995c85cd6d438de33e611eeadd63d0d4"
server_key_version: "encryptionkey@0"
I20260928 06:39:34.583657 26134 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 06:39:34.586040 26149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:34.587301 26134 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:39:34.587472 26134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "1698231d854a4094b87635706b8245d2"
format_stamp: "Formatted at 2026-09-28 06:39:34 on dist-test-slave-j0qf"
server_key: "7728c44c1f8134e7488ad5b71bb49cba"
server_key_iv: "995c85cd6d438de33e611eeadd63d0d4"
server_key_version: "encryptionkey@0"
I20260928 06:39:34.587597 26134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:34.592247 26134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:34.595286 26134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:34.595487 26134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:34.602419 26134 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34999
I20260928 06:39:34.602414 26201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34999 every 8 connection(s)
I20260928 06:39:34.603338 26134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:34.605981 26202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:34.610438 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 26134
I20260928 06:39:34.610587 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:34.610896 5208 external_mini_cluster.cc:1428] Setting key 5d02ee6635ab1ecd62a0ff9d319eb690
I20260928 06:39:34.611997 26202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1698231d854a4094b87635706b8245d2: Bootstrap starting.
I20260928 06:39:34.614435 26202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1698231d854a4094b87635706b8245d2: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:34.615469 26202 log.cc:824] T 00000000000000000000000000000000 P 1698231d854a4094b87635706b8245d2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:34.616514 26205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:34.612466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49161 (local address 127.0.0.1:34999)
0928 06:39:34.612913 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:34.612929 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:34.612964 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:39:34.613594 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:34.613610 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:34.613898 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:34.614188 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:34.615052 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:34.615056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:34.615067 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:34.615077 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:34.615127 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:34.615145 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:34.615237 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:34.615390 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 06:39:34.615559 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":159,"threads_started":1}
I20260928 06:39:34.617516 26202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1698231d854a4094b87635706b8245d2: No bootstrap required, opened a new log
I20260928 06:39:34.619414 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:34.621965 26202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1698231d854a4094b87635706b8245d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1698231d854a4094b87635706b8245d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34999 } }
I20260928 06:39:34.622262 26202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1698231d854a4094b87635706b8245d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:34.622417 26202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1698231d854a4094b87635706b8245d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1698231d854a4094b87635706b8245d2, State: Initialized, Role: FOLLOWER
I20260928 06:39:34.622923 26202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1698231d854a4094b87635706b8245d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1698231d854a4094b87635706b8245d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34999 } }
I20260928 06:39:34.623082 26202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1698231d854a4094b87635706b8245d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:34.623158 26202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1698231d854a4094b87635706b8245d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:34.623261 26202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1698231d854a4094b87635706b8245d2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:34.623718 26205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:34.621978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33887 (local address 127.0.0.1:34999)
0928 06:39:34.622118 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:34.622122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:34.622352 (+ 230us) server_negotiation.cc:407] Connection header received
0928 06:39:34.622493 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:34.622496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:34.622556 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:34.622630 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:34.623348 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:34.623352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:34.623354 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:34.623366 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:34.623398 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:34.623411 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:34.623489 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:34.623562 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 06:39:34.623602 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:39:34.624502 26202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1698231d854a4094b87635706b8245d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1698231d854a4094b87635706b8245d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34999 } }
I20260928 06:39:34.624914 26202 leader_election.cc:302] T 00000000000000000000000000000000 P 1698231d854a4094b87635706b8245d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1698231d854a4094b87635706b8245d2; no voters:
I20260928 06:39:34.625221 26202 leader_election.cc:288] T 00000000000000000000000000000000 P 1698231d854a4094b87635706b8245d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:34.625411 26213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1698231d854a4094b87635706b8245d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:34.625679 26213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1698231d854a4094b87635706b8245d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1698231d854a4094b87635706b8245d2, State: Running, Role: LEADER
I20260928 06:39:34.625996 26213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1698231d854a4094b87635706b8245d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1698231d854a4094b87635706b8245d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34999 } }
I20260928 06:39:34.626590 26202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1698231d854a4094b87635706b8245d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:34.627884 26214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1698231d854a4094b87635706b8245d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1698231d854a4094b87635706b8245d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1698231d854a4094b87635706b8245d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34999 } } }
I20260928 06:39:34.627848 26215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1698231d854a4094b87635706b8245d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1698231d854a4094b87635706b8245d2. Latest consensus state: current_term: 1 leader_uuid: "1698231d854a4094b87635706b8245d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1698231d854a4094b87635706b8245d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34999 } } }
I20260928 06:39:34.627995 26215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1698231d854a4094b87635706b8245d2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:34.627995 26214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1698231d854a4094b87635706b8245d2 [sys.catalog]: This master's current role is: LEADER
W20260928 06:39:34.631824 26228 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1698231d854a4094b87635706b8245d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:39:34.631912 26228 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:39:34.632004 26223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:34.632752 26223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:34.638562 26223 catalog_manager.cc:1347] Generated new cluster ID: 3ee7ac2dc7904f9b8fc8ac9939095f0c
I20260928 06:39:34.638631 26223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:34.658684 26223 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:34.659029 26223 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:34.666054 26223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1698231d854a4094b87635706b8245d2: Generated new TSK 0
I20260928 06:39:34.666879 26223 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:34.675747 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 26134
2026-09-28T06:39:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:34 dist-test-slave-j0qf krb5kdc[26234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:34 dist-test-slave-j0qf krb5kdc[26234](info): set up 2 sockets
Sep 28 06:39:34 dist-test-slave-j0qf krb5kdc[26234](info): commencing operation
krb5kdc: starting...
W20260928 06:39:35.725240 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:39:35 dist-test-slave-j0qf krb5kdc[26234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:39:35.854103 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40805
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:35.984663 26247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:35.985045 26247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:35.985131 26247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:35.989377 26247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:35.989470 26247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:35.989509 26247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:35.989566 26247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:35.989603 26247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:35.995292 26247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:35.996507 26247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:35.997628 26247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:36.006789 26255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:36.006815 26253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:36.006855 26252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:36.007093 26247 server_base.cc:1034] running on GCE node
I20260928 06:39:36.007650 26247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:36.008906 26247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:36.010075 26247 hybrid_clock.cc:648] HybridClock initialized: now 1790577576010061 us; error 33 us; skew 500 ppm
Sep 28 06:39:36 dist-test-slave-j0qf krb5kdc[26234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:36.013002 26247 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:39:36.013877 26247 webserver.cc:460] Webserver started at http://127.0.0.1:40333/ using document root <none> and password file <none>
I20260928 06:39:36.014469 26247 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:36.014536 26247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:36.014706 26247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:36.016721 26247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "232a12aa6af142efbc38ad4a4734e670"
format_stamp: "Formatted at 2026-09-28 06:39:36 on dist-test-slave-j0qf"
server_key: "586f3dcfa2d5e32562001aba23fb359b"
server_key_iv: "ee0928b823e55b206bdf691a3101973b"
server_key_version: "encryptionkey@0"
I20260928 06:39:36.017256 26247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "232a12aa6af142efbc38ad4a4734e670"
format_stamp: "Formatted at 2026-09-28 06:39:36 on dist-test-slave-j0qf"
server_key: "586f3dcfa2d5e32562001aba23fb359b"
server_key_iv: "ee0928b823e55b206bdf691a3101973b"
server_key_version: "encryptionkey@0"
I20260928 06:39:36.021450 26247 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 06:39:36.024290 26262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:36.025628 26247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:39:36.025770 26247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "232a12aa6af142efbc38ad4a4734e670"
format_stamp: "Formatted at 2026-09-28 06:39:36 on dist-test-slave-j0qf"
server_key: "586f3dcfa2d5e32562001aba23fb359b"
server_key_iv: "ee0928b823e55b206bdf691a3101973b"
server_key_version: "encryptionkey@0"
I20260928 06:39:36.025899 26247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:36.048025 26247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:36.051394 26247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:36.051602 26247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:36.059705 26247 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40805
I20260928 06:39:36.059722 26314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40805 every 8 connection(s)
I20260928 06:39:36.060757 26247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:36.061553 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 26247
I20260928 06:39:36.061722 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:36.062037 5208 external_mini_cluster.cc:1428] Setting key 724517e588ffc90f482a309009d11fb1
I20260928 06:39:36.064335 26315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:36.070075 26315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 232a12aa6af142efbc38ad4a4734e670: Bootstrap starting.
Sep 28 06:39:36 dist-test-slave-j0qf krb5kdc[26234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:39:36.072805 26315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 232a12aa6af142efbc38ad4a4734e670: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:36.073558 26315 log.cc:824] T 00000000000000000000000000000000 P 232a12aa6af142efbc38ad4a4734e670: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:36.076108 26315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 232a12aa6af142efbc38ad4a4734e670: No bootstrap required, opened a new log
I20260928 06:39:36.077131 26318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:36.063498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:47635 (local address 127.0.0.1:40805)
0928 06:39:36.064028 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:36.064043 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:36.064081 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:39:36.064820 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:36.064838 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:36.065188 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:36.065538 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:36.066412 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:36.067417 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:36.068283 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:36.068778 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:36.068956 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:36.071830 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:36.071835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:36.071849 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:36.071877 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:36.073726 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:36.074395 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:36.074403 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:36.074410 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:36.074560 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:36.074900 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:36.074903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:36.074904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:36.075421 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:36.075638 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:36.075911 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 06:39:36.076151 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":173,"threads_started":1}
I20260928 06:39:36.080340 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:36.080305 26315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 232a12aa6af142efbc38ad4a4734e670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "232a12aa6af142efbc38ad4a4734e670" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40805 } }
I20260928 06:39:36.080605 26315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 232a12aa6af142efbc38ad4a4734e670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:36.080683 26315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 232a12aa6af142efbc38ad4a4734e670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 232a12aa6af142efbc38ad4a4734e670, State: Initialized, Role: FOLLOWER
I20260928 06:39:36.081195 26315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 232a12aa6af142efbc38ad4a4734e670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "232a12aa6af142efbc38ad4a4734e670" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40805 } }
I20260928 06:39:36.081372 26315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 232a12aa6af142efbc38ad4a4734e670 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:36.081440 26315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 232a12aa6af142efbc38ad4a4734e670 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:36.081542 26315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 232a12aa6af142efbc38ad4a4734e670 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:36.082952 26315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 232a12aa6af142efbc38ad4a4734e670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "232a12aa6af142efbc38ad4a4734e670" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40805 } }
I20260928 06:39:36.083386 26315 leader_election.cc:302] T 00000000000000000000000000000000 P 232a12aa6af142efbc38ad4a4734e670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 232a12aa6af142efbc38ad4a4734e670; no voters:
I20260928 06:39:36.083784 26315 leader_election.cc:288] T 00000000000000000000000000000000 P 232a12aa6af142efbc38ad4a4734e670 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:36.083849 26325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 232a12aa6af142efbc38ad4a4734e670 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:36.084038 26325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 232a12aa6af142efbc38ad4a4734e670 [term 1 LEADER]: Becoming Leader. State: Replica: 232a12aa6af142efbc38ad4a4734e670, State: Running, Role: LEADER
I20260928 06:39:36.084349 26325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 232a12aa6af142efbc38ad4a4734e670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "232a12aa6af142efbc38ad4a4734e670" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40805 } }
I20260928 06:39:36.085135 26315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 232a12aa6af142efbc38ad4a4734e670 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:36.086088 26326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 232a12aa6af142efbc38ad4a4734e670 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 232a12aa6af142efbc38ad4a4734e670. Latest consensus state: current_term: 1 leader_uuid: "232a12aa6af142efbc38ad4a4734e670" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "232a12aa6af142efbc38ad4a4734e670" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40805 } } }
I20260928 06:39:36.086231 26326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 232a12aa6af142efbc38ad4a4734e670 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:36.086511 26327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 232a12aa6af142efbc38ad4a4734e670 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "232a12aa6af142efbc38ad4a4734e670" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "232a12aa6af142efbc38ad4a4734e670" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40805 } } }
I20260928 06:39:36.086620 26327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 232a12aa6af142efbc38ad4a4734e670 [sys.catalog]: This master's current role is: LEADER
W20260928 06:39:36.090780 26340 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 232a12aa6af142efbc38ad4a4734e670: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:39:36.090940 26340 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:39:36.091027 26334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:36.091971 26334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:36.092947 26318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:36.082800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:46673 (local address 127.0.0.1:40805)
0928 06:39:36.082951 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:36.082957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:36.083147 (+ 190us) server_negotiation.cc:407] Connection header received
0928 06:39:36.083354 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:36.083358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:36.083417 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:36.083512 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:36.084361 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:36.085030 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:36.086253 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:36.086620 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:36.086749 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:36.088880 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:36.088884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:36.088887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:36.088914 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:36.090739 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:36.091376 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:36.091379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:36.091381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:36.091475 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:36.091879 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:36.091889 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:36.091891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:36.092144 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:36.092305 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:36.092671 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 06:39:36.092781 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 06:39:36.098425 26334 catalog_manager.cc:1347] Generated new cluster ID: 48046d4e472143528ca1c5c677bbdcb9
I20260928 06:39:36.098505 26334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:36.116014 26334 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:36.116957 26334 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:36.120419 26334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 232a12aa6af142efbc38ad4a4734e670: Generated new TSK 0
I20260928 06:39:36.121196 26334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:36.127712 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 26247
W20260928 06:39:36.133782 26232 connection.cc:535] client connection to 127.0.0.1:40805 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40805): Connection reset by peer (error 104)
2026-09-28T06:39:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1452 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:36 dist-test-slave-j0qf krb5kdc[26346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:36 dist-test-slave-j0qf krb5kdc[26346](info): set up 2 sockets
Sep 28 06:39:36 dist-test-slave-j0qf krb5kdc[26346](info): commencing operation
krb5kdc: starting...
W20260928 06:39:37.169147 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:39:37 dist-test-slave-j0qf krb5kdc[26346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:39:37.290175 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40151
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:37.398430 26360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:37.398695 26360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:37.398777 26360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:37.402840 26360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:37.402909 26360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:37.402928 26360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:37.402974 26360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:37.403012 26360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:37.408401 26360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:37.409533 26360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:37.410629 26360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:37.419278 26366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:37.419296 26368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:37.419406 26365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:37.419638 26360 server_base.cc:1034] running on GCE node
I20260928 06:39:37.420297 26360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:37.421444 26360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:37.422631 26360 hybrid_clock.cc:648] HybridClock initialized: now 1790577577422619 us; error 38 us; skew 500 ppm
Sep 28 06:39:37 dist-test-slave-j0qf krb5kdc[26346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:37.425341 26360 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:39:37.426141 26360 webserver.cc:460] Webserver started at http://127.0.0.1:42791/ using document root <none> and password file <none>
I20260928 06:39:37.426703 26360 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:37.426765 26360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:37.426927 26360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:37.428864 26360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "2da45943e6ac4cfaadc9d6f1483c598a"
format_stamp: "Formatted at 2026-09-28 06:39:37 on dist-test-slave-j0qf"
server_key: "8a6847561a9669ba1c3ea61cfd43c40f"
server_key_iv: "480d90ac92c40f433dadef2c38a1ed5c"
server_key_version: "encryptionkey@0"
I20260928 06:39:37.429365 26360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "2da45943e6ac4cfaadc9d6f1483c598a"
format_stamp: "Formatted at 2026-09-28 06:39:37 on dist-test-slave-j0qf"
server_key: "8a6847561a9669ba1c3ea61cfd43c40f"
server_key_iv: "480d90ac92c40f433dadef2c38a1ed5c"
server_key_version: "encryptionkey@0"
I20260928 06:39:37.432960 26360 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 06:39:37.435374 26375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:37.436594 26360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:39:37.436690 26360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "2da45943e6ac4cfaadc9d6f1483c598a"
format_stamp: "Formatted at 2026-09-28 06:39:37 on dist-test-slave-j0qf"
server_key: "8a6847561a9669ba1c3ea61cfd43c40f"
server_key_iv: "480d90ac92c40f433dadef2c38a1ed5c"
server_key_version: "encryptionkey@0"
I20260928 06:39:37.436776 26360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:37.458541 26360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:37.461447 26360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:37.461601 26360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:37.468786 26360 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40151
I20260928 06:39:37.468808 26427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40151 every 8 connection(s)
I20260928 06:39:37.469769 26360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:37.472478 26428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:37.475991 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 26360
I20260928 06:39:37.476135 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:37.476403 5208 external_mini_cluster.cc:1428] Setting key a0426d7c30bc439036148c36d769ee25
I20260928 06:39:37.478415 26428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2da45943e6ac4cfaadc9d6f1483c598a: Bootstrap starting.
I20260928 06:39:37.481130 26428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2da45943e6ac4cfaadc9d6f1483c598a: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:37.482051 26428 log.cc:824] T 00000000000000000000000000000000 P 2da45943e6ac4cfaadc9d6f1483c598a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:37.484261 26428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2da45943e6ac4cfaadc9d6f1483c598a: No bootstrap required, opened a new log
I20260928 06:39:37.486575 26431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:37.477833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:55249 (local address 127.0.0.1:40151)
0928 06:39:37.478254 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:37.478273 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:37.478354 (+ 81us) server_negotiation.cc:407] Connection header received
0928 06:39:37.479151 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:37.479173 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:37.479622 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:37.479986 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:37.480397 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:37.481859 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:37.483127 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:37.483791 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:37.483961 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:37.484784 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:37.484789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:37.484803 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:37.485189 (+ 386us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:37.485245 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:37.485268 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:37.485376 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:37.485617 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 06:39:37.485767 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":113,"threads_started":1}
I20260928 06:39:37.488296 26428 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2da45943e6ac4cfaadc9d6f1483c598a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da45943e6ac4cfaadc9d6f1483c598a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40151 } }
I20260928 06:39:37.488600 26428 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2da45943e6ac4cfaadc9d6f1483c598a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:37.488667 26428 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2da45943e6ac4cfaadc9d6f1483c598a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2da45943e6ac4cfaadc9d6f1483c598a, State: Initialized, Role: FOLLOWER
I20260928 06:39:37.489131 26428 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2da45943e6ac4cfaadc9d6f1483c598a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da45943e6ac4cfaadc9d6f1483c598a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40151 } }
I20260928 06:39:37.489269 26428 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2da45943e6ac4cfaadc9d6f1483c598a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:37.489351 26428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2da45943e6ac4cfaadc9d6f1483c598a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:37.489454 26428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2da45943e6ac4cfaadc9d6f1483c598a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:37.489699 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:37.490439 26428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2da45943e6ac4cfaadc9d6f1483c598a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da45943e6ac4cfaadc9d6f1483c598a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40151 } }
I20260928 06:39:37.490759 26428 leader_election.cc:302] T 00000000000000000000000000000000 P 2da45943e6ac4cfaadc9d6f1483c598a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2da45943e6ac4cfaadc9d6f1483c598a; no voters:
I20260928 06:39:37.491091 26428 leader_election.cc:288] T 00000000000000000000000000000000 P 2da45943e6ac4cfaadc9d6f1483c598a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:37.491195 26436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2da45943e6ac4cfaadc9d6f1483c598a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:37.491386 26436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2da45943e6ac4cfaadc9d6f1483c598a [term 1 LEADER]: Becoming Leader. State: Replica: 2da45943e6ac4cfaadc9d6f1483c598a, State: Running, Role: LEADER
I20260928 06:39:37.491675 26436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2da45943e6ac4cfaadc9d6f1483c598a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da45943e6ac4cfaadc9d6f1483c598a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40151 } }
I20260928 06:39:37.492362 26428 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2da45943e6ac4cfaadc9d6f1483c598a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:37.492815 26439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2da45943e6ac4cfaadc9d6f1483c598a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2da45943e6ac4cfaadc9d6f1483c598a. Latest consensus state: current_term: 1 leader_uuid: "2da45943e6ac4cfaadc9d6f1483c598a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da45943e6ac4cfaadc9d6f1483c598a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40151 } } }
I20260928 06:39:37.492929 26439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2da45943e6ac4cfaadc9d6f1483c598a [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:37.493616 26438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2da45943e6ac4cfaadc9d6f1483c598a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2da45943e6ac4cfaadc9d6f1483c598a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da45943e6ac4cfaadc9d6f1483c598a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40151 } } }
I20260928 06:39:37.493706 26438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2da45943e6ac4cfaadc9d6f1483c598a [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:37.494354 26442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:37.496953 26431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:37.491863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:41953 (local address 127.0.0.1:40151)
0928 06:39:37.491959 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:37.491963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:37.492460 (+ 497us) server_negotiation.cc:407] Connection header received
0928 06:39:37.492623 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:37.492625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:37.492669 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:37.492731 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:37.493060 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:37.493806 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:37.495057 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:37.495405 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:37.495535 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:37.496460 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:37.496463 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:37.496466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:37.496479 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:37.496506 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:37.496519 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:37.496610 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:37.496780 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 06:39:37.496813 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:39:37.498116 26442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:37.503918 26442 catalog_manager.cc:1347] Generated new cluster ID: 0ebb894137f742bdbf01aa9c920cb5da
I20260928 06:39:37.503986 26442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:37.522781 26442 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:37.523731 26442 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:37.534659 26442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2da45943e6ac4cfaadc9d6f1483c598a: Generated new TSK 0
I20260928 06:39:37.535400 26442 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:37.546960 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 26360
W20260928 06:39:37.552681 26344 connection.cc:535] client connection to 127.0.0.1:40151 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40151): Connection reset by peer (error 104)
2026-09-28T06:39:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:37 dist-test-slave-j0qf krb5kdc[26459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:37 dist-test-slave-j0qf krb5kdc[26459](info): set up 2 sockets
Sep 28 06:39:37 dist-test-slave-j0qf krb5kdc[26459](info): commencing operation
krb5kdc: starting...
W20260928 06:39:38.616869 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:39:38 dist-test-slave-j0qf krb5kdc[26459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:39:38.741070 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34925
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:38.852806 26472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:38.853147 26472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:38.853191 26472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:38.857337 26472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:38.857400 26472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:38.857419 26472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:38.857436 26472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:38.857458 26472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:38.862929 26472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:38.863981 26472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:38.864988 26472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:38.873974 26480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:38.873978 26478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:38.874003 26477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:38.874029 26472 server_base.cc:1034] running on GCE node
I20260928 06:39:38.874804 26472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:38.875958 26472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:38.877130 26472 hybrid_clock.cc:648] HybridClock initialized: now 1790577578877114 us; error 49 us; skew 500 ppm
Sep 28 06:39:38 dist-test-slave-j0qf krb5kdc[26459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:38.879956 26472 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:39:38.880926 26472 webserver.cc:460] Webserver started at http://127.0.0.1:36149/ using document root <none> and password file <none>
I20260928 06:39:38.881501 26472 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:38.881564 26472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:38.881783 26472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:38.883949 26472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "7dcd5696625640cfb8763f62f95ca92e"
format_stamp: "Formatted at 2026-09-28 06:39:38 on dist-test-slave-j0qf"
server_key: "c71e49c6f82e6a77b9ae1dbde8963d1f"
server_key_iv: "7240b37a7173b9e8e7e134cb09a50dda"
server_key_version: "encryptionkey@0"
I20260928 06:39:38.884521 26472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "7dcd5696625640cfb8763f62f95ca92e"
format_stamp: "Formatted at 2026-09-28 06:39:38 on dist-test-slave-j0qf"
server_key: "c71e49c6f82e6a77b9ae1dbde8963d1f"
server_key_iv: "7240b37a7173b9e8e7e134cb09a50dda"
server_key_version: "encryptionkey@0"
I20260928 06:39:38.888233 26472 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260928 06:39:38.890647 26487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:38.891880 26472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 06:39:38.892133 26472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "7dcd5696625640cfb8763f62f95ca92e"
format_stamp: "Formatted at 2026-09-28 06:39:38 on dist-test-slave-j0qf"
server_key: "c71e49c6f82e6a77b9ae1dbde8963d1f"
server_key_iv: "7240b37a7173b9e8e7e134cb09a50dda"
server_key_version: "encryptionkey@0"
I20260928 06:39:38.892256 26472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:38.896828 26472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:38.899843 26472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:38.900046 26472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:38.907423 26472 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34925
I20260928 06:39:38.907464 26540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34925 every 8 connection(s)
I20260928 06:39:38.908358 26472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:38.911106 26541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:38.916610 26541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dcd5696625640cfb8763f62f95ca92e: Bootstrap starting.
I20260928 06:39:38.917034 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 26472
I20260928 06:39:38.917184 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:38.917485 5208 external_mini_cluster.cc:1428] Setting key ed3463ecd204405d93843797c2bc1735
I20260928 06:39:38.919585 26541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dcd5696625640cfb8763f62f95ca92e: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:38.920372 26541 log.cc:824] T 00000000000000000000000000000000 P 7dcd5696625640cfb8763f62f95ca92e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:38.922731 26541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dcd5696625640cfb8763f62f95ca92e: No bootstrap required, opened a new log
I20260928 06:39:38.922910 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:38.918847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:43527 (local address 127.0.0.1:34925)
0928 06:39:38.919279 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:38.919294 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:38.919331 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:39:38.920048 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:38.920087 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:38.920378 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:38.920708 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:38.921547 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:38.921552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:38.921563 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:38.921572 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:38.921624 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:38.921643 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:38.921730 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:38.921878 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 06:39:38.922073 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":132,"threads_started":1}
I20260928 06:39:38.925889 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:38.926877 26541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dcd5696625640cfb8763f62f95ca92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcd5696625640cfb8763f62f95ca92e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34925 } }
I20260928 06:39:38.927139 26541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dcd5696625640cfb8763f62f95ca92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:38.927217 26541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dcd5696625640cfb8763f62f95ca92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dcd5696625640cfb8763f62f95ca92e, State: Initialized, Role: FOLLOWER
I20260928 06:39:38.927716 26541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dcd5696625640cfb8763f62f95ca92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcd5696625640cfb8763f62f95ca92e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34925 } }
I20260928 06:39:38.927870 26541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dcd5696625640cfb8763f62f95ca92e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:38.927959 26541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dcd5696625640cfb8763f62f95ca92e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:38.928059 26541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dcd5696625640cfb8763f62f95ca92e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:38.929200 26541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dcd5696625640cfb8763f62f95ca92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcd5696625640cfb8763f62f95ca92e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34925 } }
I20260928 06:39:38.929602 26541 leader_election.cc:302] T 00000000000000000000000000000000 P 7dcd5696625640cfb8763f62f95ca92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dcd5696625640cfb8763f62f95ca92e; no voters:
I20260928 06:39:38.929833 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:38.928182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:44141 (local address 127.0.0.1:34925)
0928 06:39:38.928311 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:38.928318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:38.928514 (+ 196us) server_negotiation.cc:407] Connection header received
0928 06:39:38.928594 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:38.928597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:38.928631 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:38.928713 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:38.929478 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:38.929482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:38.929485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:38.929497 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:38.929527 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:38.929540 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:38.929636 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:38.929671 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 06:39:38.929723 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:39:38.929963 26541 leader_election.cc:288] T 00000000000000000000000000000000 P 7dcd5696625640cfb8763f62f95ca92e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:38.930311 26552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dcd5696625640cfb8763f62f95ca92e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:38.930531 26552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dcd5696625640cfb8763f62f95ca92e [term 1 LEADER]: Becoming Leader. State: Replica: 7dcd5696625640cfb8763f62f95ca92e, State: Running, Role: LEADER
I20260928 06:39:38.930833 26552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dcd5696625640cfb8763f62f95ca92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcd5696625640cfb8763f62f95ca92e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34925 } }
I20260928 06:39:38.931336 26541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dcd5696625640cfb8763f62f95ca92e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:38.932548 26553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dcd5696625640cfb8763f62f95ca92e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dcd5696625640cfb8763f62f95ca92e. Latest consensus state: current_term: 1 leader_uuid: "7dcd5696625640cfb8763f62f95ca92e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcd5696625640cfb8763f62f95ca92e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34925 } } }
I20260928 06:39:38.932951 26553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dcd5696625640cfb8763f62f95ca92e [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:38.932842 26551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dcd5696625640cfb8763f62f95ca92e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dcd5696625640cfb8763f62f95ca92e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcd5696625640cfb8763f62f95ca92e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34925 } } }
I20260928 06:39:38.933276 26551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dcd5696625640cfb8763f62f95ca92e [sys.catalog]: This master's current role is: LEADER
W20260928 06:39:38.935911 26566 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7dcd5696625640cfb8763f62f95ca92e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:39:38.935976 26566 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:39:38.936101 26564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:38.936844 26564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:38.942088 26564 catalog_manager.cc:1347] Generated new cluster ID: 3aed183f03e940579b67a1be3cf65999
I20260928 06:39:38.942160 26564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:38.961426 26564 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:38.961812 26564 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:38.966668 26564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dcd5696625640cfb8763f62f95ca92e: Generated new TSK 0
I20260928 06:39:38.967443 26564 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:38.975633 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 26472
2026-09-28T06:39:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1428 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:39 dist-test-slave-j0qf krb5kdc[26572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:39 dist-test-slave-j0qf krb5kdc[26572](info): set up 2 sockets
Sep 28 06:39:39 dist-test-slave-j0qf krb5kdc[26572](info): commencing operation
krb5kdc: starting...
W20260928 06:39:40.097440 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:39:40 dist-test-slave-j0qf krb5kdc[26572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:39:40.226491 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38817
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:40.347388 26585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:40.347687 26585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:40.347760 26585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:40.351863 26585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:40.351951 26585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:40.351989 26585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:40.352042 26585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:40.352080 26585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:40.357640 26585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38817
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:40.359059 26585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:40.360307 26585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:40.369367 26590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:40.369367 26593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:40.369423 26585 server_base.cc:1034] running on GCE node
W20260928 06:39:40.369401 26591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:40.370277 26585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:40.371639 26585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:40.372804 26585 hybrid_clock.cc:648] HybridClock initialized: now 1790577580372778 us; error 44 us; skew 500 ppm
Sep 28 06:39:40 dist-test-slave-j0qf krb5kdc[26572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:40.375830 26585 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:39:40.376825 26585 webserver.cc:460] Webserver started at http://127.0.0.1:43967/ using document root <none> and password file <none>
I20260928 06:39:40.377409 26585 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:40.377527 26585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:40.377774 26585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:40.379951 26585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "8816cb739a0b4167b1945a08a6752a54"
format_stamp: "Formatted at 2026-09-28 06:39:40 on dist-test-slave-j0qf"
server_key: "86432386fa4aa58db600e557e0804618"
server_key_iv: "f7f48c7b9ac0af4dba89ef22d70e2467"
server_key_version: "encryptionkey@0"
I20260928 06:39:40.380523 26585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "8816cb739a0b4167b1945a08a6752a54"
format_stamp: "Formatted at 2026-09-28 06:39:40 on dist-test-slave-j0qf"
server_key: "86432386fa4aa58db600e557e0804618"
server_key_iv: "f7f48c7b9ac0af4dba89ef22d70e2467"
server_key_version: "encryptionkey@0"
I20260928 06:39:40.384593 26585 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260928 06:39:40.387108 26600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:40.388419 26585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 06:39:40.388566 26585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "8816cb739a0b4167b1945a08a6752a54"
format_stamp: "Formatted at 2026-09-28 06:39:40 on dist-test-slave-j0qf"
server_key: "86432386fa4aa58db600e557e0804618"
server_key_iv: "f7f48c7b9ac0af4dba89ef22d70e2467"
server_key_version: "encryptionkey@0"
I20260928 06:39:40.388684 26585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:40.403472 26585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:40.406838 26585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:40.407075 26585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:40.415385 26585 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38817
I20260928 06:39:40.415412 26652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38817 every 8 connection(s)
I20260928 06:39:40.416402 26585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:40.419167 26653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:40.423851 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 26585
I20260928 06:39:40.424018 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:40.424558 5208 external_mini_cluster.cc:1428] Setting key ac6909acd0608fa79c2acf7dcaaa6c32
I20260928 06:39:40.425572 26653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8816cb739a0b4167b1945a08a6752a54: Bootstrap starting.
I20260928 06:39:40.428189 26653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8816cb739a0b4167b1945a08a6752a54: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:40.428972 26653 log.cc:824] T 00000000000000000000000000000000 P 8816cb739a0b4167b1945a08a6752a54: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:40.430934 26653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8816cb739a0b4167b1945a08a6752a54: No bootstrap required, opened a new log
Sep 28 06:39:40 dist-test-slave-j0qf krb5kdc[26572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:39:40.434958 26653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8816cb739a0b4167b1945a08a6752a54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8816cb739a0b4167b1945a08a6752a54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38817 } }
I20260928 06:39:40.435169 26653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8816cb739a0b4167b1945a08a6752a54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:40.435230 26653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8816cb739a0b4167b1945a08a6752a54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8816cb739a0b4167b1945a08a6752a54, State: Initialized, Role: FOLLOWER
I20260928 06:39:40.435716 26653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8816cb739a0b4167b1945a08a6752a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8816cb739a0b4167b1945a08a6752a54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38817 } }
I20260928 06:39:40.435844 26653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8816cb739a0b4167b1945a08a6752a54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:40.435930 26653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8816cb739a0b4167b1945a08a6752a54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:40.436041 26653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8816cb739a0b4167b1945a08a6752a54 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:40.436901 26653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8816cb739a0b4167b1945a08a6752a54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8816cb739a0b4167b1945a08a6752a54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38817 } }
I20260928 06:39:40.437237 26653 leader_election.cc:302] T 00000000000000000000000000000000 P 8816cb739a0b4167b1945a08a6752a54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8816cb739a0b4167b1945a08a6752a54; no voters:
I20260928 06:39:40.437609 26653 leader_election.cc:288] T 00000000000000000000000000000000 P 8816cb739a0b4167b1945a08a6752a54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:40.437680 26658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8816cb739a0b4167b1945a08a6752a54 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:40.437904 26658 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8816cb739a0b4167b1945a08a6752a54 [term 1 LEADER]: Becoming Leader. State: Replica: 8816cb739a0b4167b1945a08a6752a54, State: Running, Role: LEADER
I20260928 06:39:40.438201 26658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8816cb739a0b4167b1945a08a6752a54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8816cb739a0b4167b1945a08a6752a54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38817 } }
I20260928 06:39:40.438975 26653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8816cb739a0b4167b1945a08a6752a54 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:40.440157 26659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8816cb739a0b4167b1945a08a6752a54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8816cb739a0b4167b1945a08a6752a54. Latest consensus state: current_term: 1 leader_uuid: "8816cb739a0b4167b1945a08a6752a54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8816cb739a0b4167b1945a08a6752a54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38817 } } }
I20260928 06:39:40.440332 26659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8816cb739a0b4167b1945a08a6752a54 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:40.440745 26656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:40.426452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:35305 (local address 127.0.0.1:38817)
0928 06:39:40.426947 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:40.426962 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:40.426998 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:39:40.427636 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:40.427654 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:40.428006 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:40.428358 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:40.429265 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:40.430232 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:40.431306 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:40.431782 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:40.431964 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:40.434965 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:40.434970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:40.434983 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:40.435010 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:40.437015 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:40.437627 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:40.437633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:40.437638 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:40.437794 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:40.438227 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:40.438232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:40.438234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:40.438756 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:40.439022 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:40.439343 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 06:39:40.439587 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":179,"threads_started":1}
I20260928 06:39:40.441119 26668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:40.441677 26660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8816cb739a0b4167b1945a08a6752a54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8816cb739a0b4167b1945a08a6752a54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8816cb739a0b4167b1945a08a6752a54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38817 } } }
I20260928 06:39:40.441927 26660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8816cb739a0b4167b1945a08a6752a54 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:40.444322 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:40.444772 26668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:40.451512 26668 catalog_manager.cc:1347] Generated new cluster ID: 852099c9183e41f2b4b04c162ddeb146
I20260928 06:39:40.451576 26668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:40.456679 26656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:40.447843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:47145 (local address 127.0.0.1:38817)
0928 06:39:40.447948 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:40.447951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:40.448244 (+ 293us) server_negotiation.cc:407] Connection header received
0928 06:39:40.448365 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:40.448368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:40.448411 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:40.448502 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:40.449237 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:40.449953 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:40.451180 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:40.451557 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:40.451637 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:40.453187 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:40.453191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:40.453193 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:40.453218 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:40.454846 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:40.455332 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:40.455336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:40.455337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:40.455421 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:40.455759 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:40.455762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:40.455764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:40.455946 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:40.456102 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:40.456378 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 06:39:40.456473 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:39:40.467962 26668 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:40.469002 26668 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:40.475528 26668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8816cb739a0b4167b1945a08a6752a54: Generated new TSK 0
I20260928 06:39:40.476251 26668 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:40.484474 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 26585
W20260928 06:39:40.490177 26570 connection.cc:535] client connection to 127.0.0.1:38817 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38817): Connection reset by peer (error 104)
2026-09-28T06:39:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1508 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:40 dist-test-slave-j0qf krb5kdc[26685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:40 dist-test-slave-j0qf krb5kdc[26685](info): set up 2 sockets
Sep 28 06:39:40 dist-test-slave-j0qf krb5kdc[26685](info): commencing operation
krb5kdc: starting...
W20260928 06:39:41.533763 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:39:41 dist-test-slave-j0qf krb5kdc[26685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:39:41.661100 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44233
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:41.769727 26699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:41.770052 26699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:41.770138 26699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:41.774292 26699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:41.774392 26699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:41.774430 26699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:41.774485 26699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:41.774524 26699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:41.779752 26699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44233
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:41.780819 26699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:41.781849 26699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:41.790369 26705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:41.790369 26704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:41.790364 26707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:41.790722 26699 server_base.cc:1034] running on GCE node
I20260928 06:39:41.791306 26699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:41.792404 26699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:41.793596 26699 hybrid_clock.cc:648] HybridClock initialized: now 1790577581793582 us; error 31 us; skew 500 ppm
Sep 28 06:39:41 dist-test-slave-j0qf krb5kdc[26685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:41.796397 26699 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:39:41.797199 26699 webserver.cc:460] Webserver started at http://127.0.0.1:38197/ using document root <none> and password file <none>
I20260928 06:39:41.797710 26699 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:41.797770 26699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:41.797930 26699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:41.800387 26699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "05831d01a20c42009f402e14f0d60e00"
format_stamp: "Formatted at 2026-09-28 06:39:41 on dist-test-slave-j0qf"
server_key: "7d079366464959f31cdbfed310872b97"
server_key_iv: "580d8068990230ed6b8df8aafb1e1919"
server_key_version: "encryptionkey@0"
I20260928 06:39:41.801085 26699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "05831d01a20c42009f402e14f0d60e00"
format_stamp: "Formatted at 2026-09-28 06:39:41 on dist-test-slave-j0qf"
server_key: "7d079366464959f31cdbfed310872b97"
server_key_iv: "580d8068990230ed6b8df8aafb1e1919"
server_key_version: "encryptionkey@0"
I20260928 06:39:41.805436 26699 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260928 06:39:41.808012 26714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:41.809279 26699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260928 06:39:41.809453 26699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "05831d01a20c42009f402e14f0d60e00"
format_stamp: "Formatted at 2026-09-28 06:39:41 on dist-test-slave-j0qf"
server_key: "7d079366464959f31cdbfed310872b97"
server_key_iv: "580d8068990230ed6b8df8aafb1e1919"
server_key_version: "encryptionkey@0"
I20260928 06:39:41.809574 26699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:41.820703 26699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:41.823748 26699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:41.823971 26699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:41.831426 26699 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44233
I20260928 06:39:41.831446 26766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44233 every 8 connection(s)
I20260928 06:39:41.832432 26699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:41.835197 26768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:41.837267 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 26699
I20260928 06:39:41.837440 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:41.837733 5208 external_mini_cluster.cc:1428] Setting key 572db94c6c6373d936f1d4f93aad01bd
I20260928 06:39:41.841359 26768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05831d01a20c42009f402e14f0d60e00: Bootstrap starting.
I20260928 06:39:41.843662 26768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05831d01a20c42009f402e14f0d60e00: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:41.844411 26768 log.cc:824] T 00000000000000000000000000000000 P 05831d01a20c42009f402e14f0d60e00: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:41.846626 26768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05831d01a20c42009f402e14f0d60e00: No bootstrap required, opened a new log
I20260928 06:39:41.847234 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:41.839252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:56055 (local address 127.0.0.1:44233)
0928 06:39:41.839711 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:41.839727 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:41.839766 (+ 39us) server_negotiation.cc:407] Connection header received
0928 06:39:41.840471 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:41.840490 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:41.840858 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:41.841208 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:41.841630 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:41.842713 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:41.843626 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:41.844163 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:41.844348 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:41.845341 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:41.845347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:41.845359 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:41.845742 (+ 383us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:41.845796 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:41.845820 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:41.845929 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:41.846162 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 06:39:41.846381 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":142,"threads_started":1}
I20260928 06:39:41.850416 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:41.850550 26768 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05831d01a20c42009f402e14f0d60e00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05831d01a20c42009f402e14f0d60e00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44233 } }
I20260928 06:39:41.850803 26768 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05831d01a20c42009f402e14f0d60e00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:41.850868 26768 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05831d01a20c42009f402e14f0d60e00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05831d01a20c42009f402e14f0d60e00, State: Initialized, Role: FOLLOWER
I20260928 06:39:41.851356 26768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05831d01a20c42009f402e14f0d60e00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05831d01a20c42009f402e14f0d60e00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44233 } }
I20260928 06:39:41.851486 26768 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05831d01a20c42009f402e14f0d60e00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:41.851569 26768 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05831d01a20c42009f402e14f0d60e00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:41.851684 26768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05831d01a20c42009f402e14f0d60e00 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:41.853176 26768 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05831d01a20c42009f402e14f0d60e00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05831d01a20c42009f402e14f0d60e00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44233 } }
I20260928 06:39:41.853615 26768 leader_election.cc:302] T 00000000000000000000000000000000 P 05831d01a20c42009f402e14f0d60e00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05831d01a20c42009f402e14f0d60e00; no voters:
I20260928 06:39:41.853997 26768 leader_election.cc:288] T 00000000000000000000000000000000 P 05831d01a20c42009f402e14f0d60e00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:41.854085 26778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05831d01a20c42009f402e14f0d60e00 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:41.854288 26778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05831d01a20c42009f402e14f0d60e00 [term 1 LEADER]: Becoming Leader. State: Replica: 05831d01a20c42009f402e14f0d60e00, State: Running, Role: LEADER
I20260928 06:39:41.854627 26778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05831d01a20c42009f402e14f0d60e00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05831d01a20c42009f402e14f0d60e00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44233 } }
I20260928 06:39:41.855494 26768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05831d01a20c42009f402e14f0d60e00 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:41.856284 26779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05831d01a20c42009f402e14f0d60e00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05831d01a20c42009f402e14f0d60e00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05831d01a20c42009f402e14f0d60e00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44233 } } }
I20260928 06:39:41.856408 26779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05831d01a20c42009f402e14f0d60e00 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:41.856648 26779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05831d01a20c42009f402e14f0d60e00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05831d01a20c42009f402e14f0d60e00. Latest consensus state: current_term: 1 leader_uuid: "05831d01a20c42009f402e14f0d60e00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05831d01a20c42009f402e14f0d60e00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44233 } } }
I20260928 06:39:41.856739 26779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05831d01a20c42009f402e14f0d60e00 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:41.857023 26787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:41.858342 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:41.852836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:58087 (local address 127.0.0.1:44233)
0928 06:39:41.852985 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:41.852993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:41.853350 (+ 357us) server_negotiation.cc:407] Connection header received
0928 06:39:41.853446 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:41.853448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:41.853492 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:41.853560 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:41.853931 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:41.854780 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:41.855915 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:41.856278 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:41.856394 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:41.857567 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:41.857572 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:41.857576 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:41.857590 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:41.857622 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:41.857641 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:41.857736 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:41.858116 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 06:39:41.858160 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 06:39:41.860304 26787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:41.866470 26787 catalog_manager.cc:1347] Generated new cluster ID: d425bb1ce6cc4cf0bf62e1d2cb76bc67
I20260928 06:39:41.866546 26787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:41.897388 26787 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:41.898694 26787 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:41.906996 26787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05831d01a20c42009f402e14f0d60e00: Generated new TSK 0
I20260928 06:39:41.907822 26787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:41.917263 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 26699
W20260928 06:39:41.923506 26683 connection.cc:535] client connection to 127.0.0.1:44233 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44233): Connection reset by peer (error 104)
2026-09-28T06:39:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:41 dist-test-slave-j0qf krb5kdc[26799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:41 dist-test-slave-j0qf krb5kdc[26799](info): set up 2 sockets
Sep 28 06:39:41 dist-test-slave-j0qf krb5kdc[26799](info): commencing operation
krb5kdc: starting...
W20260928 06:39:42.956918 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:39:43 dist-test-slave-j0qf krb5kdc[26799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:39:43.085068 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32913
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:43.195979 26812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:43.196282 26812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:43.196367 26812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:43.200465 26812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:43.200529 26812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:43.200549 26812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:43.200567 26812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:43.200588 26812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:43.206455 26812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32913
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:43.207551 26812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:43.208549 26812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:43.217304 26817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:43.217419 26818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:43.217381 26812 server_base.cc:1034] running on GCE node
W20260928 06:39:43.217320 26820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:43.218247 26812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:43.219535 26812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:43.220714 26812 hybrid_clock.cc:648] HybridClock initialized: now 1790577583220686 us; error 43 us; skew 500 ppm
Sep 28 06:39:43 dist-test-slave-j0qf krb5kdc[26799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:43.223698 26812 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:39:43.224803 26812 webserver.cc:460] Webserver started at http://127.0.0.1:41955/ using document root <none> and password file <none>
I20260928 06:39:43.225419 26812 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:43.225485 26812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:43.225664 26812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:43.227888 26812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "33283e314eb7455baa5f4d41e101a285"
format_stamp: "Formatted at 2026-09-28 06:39:43 on dist-test-slave-j0qf"
server_key: "f65b94464534df055ad34006a2d740d1"
server_key_iv: "5f8f319b0166b5a39202327a97ac3316"
server_key_version: "encryptionkey@0"
I20260928 06:39:43.228428 26812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "33283e314eb7455baa5f4d41e101a285"
format_stamp: "Formatted at 2026-09-28 06:39:43 on dist-test-slave-j0qf"
server_key: "f65b94464534df055ad34006a2d740d1"
server_key_iv: "5f8f319b0166b5a39202327a97ac3316"
server_key_version: "encryptionkey@0"
I20260928 06:39:43.232338 26812 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:39:43.234838 26827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:43.236138 26812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:39:43.236274 26812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "33283e314eb7455baa5f4d41e101a285"
format_stamp: "Formatted at 2026-09-28 06:39:43 on dist-test-slave-j0qf"
server_key: "f65b94464534df055ad34006a2d740d1"
server_key_iv: "5f8f319b0166b5a39202327a97ac3316"
server_key_version: "encryptionkey@0"
I20260928 06:39:43.236393 26812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:43.241200 26812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:43.244179 26812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:43.244382 26812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:43.251705 26812 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32913
I20260928 06:39:43.251732 26880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32913 every 8 connection(s)
I20260928 06:39:43.252696 26812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:43.255443 26881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:43.260794 26881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33283e314eb7455baa5f4d41e101a285: Bootstrap starting.
I20260928 06:39:43.261498 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 26812
I20260928 06:39:43.261641 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:43.261941 5208 external_mini_cluster.cc:1428] Setting key dc71be6c6f1ef52f70f96a2c88fd6afb
I20260928 06:39:43.263788 26881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33283e314eb7455baa5f4d41e101a285: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:43.264513 26881 log.cc:824] T 00000000000000000000000000000000 P 33283e314eb7455baa5f4d41e101a285: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:43.266827 26881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33283e314eb7455baa5f4d41e101a285: No bootstrap required, opened a new log
I20260928 06:39:43.267416 26884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:43.263273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:35475 (local address 127.0.0.1:32913)
0928 06:39:43.263737 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:43.263751 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:43.263786 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:39:43.264495 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:43.264512 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:43.264885 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:43.265262 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:43.266087 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:43.266092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:43.266104 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:43.266113 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:43.266165 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:43.266184 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:43.266270 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:43.266435 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 06:39:43.266616 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":140,"threads_started":1}
I20260928 06:39:43.270236 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:43.270527 26881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33283e314eb7455baa5f4d41e101a285 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33283e314eb7455baa5f4d41e101a285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32913 } }
I20260928 06:39:43.270816 26881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33283e314eb7455baa5f4d41e101a285 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:43.270876 26881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33283e314eb7455baa5f4d41e101a285 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33283e314eb7455baa5f4d41e101a285, State: Initialized, Role: FOLLOWER
I20260928 06:39:43.271365 26881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33283e314eb7455baa5f4d41e101a285 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33283e314eb7455baa5f4d41e101a285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32913 } }
I20260928 06:39:43.271526 26881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33283e314eb7455baa5f4d41e101a285 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:43.271602 26881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33283e314eb7455baa5f4d41e101a285 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:43.271721 26881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33283e314eb7455baa5f4d41e101a285 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:43.273010 26881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33283e314eb7455baa5f4d41e101a285 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33283e314eb7455baa5f4d41e101a285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32913 } }
I20260928 06:39:43.273396 26881 leader_election.cc:302] T 00000000000000000000000000000000 P 33283e314eb7455baa5f4d41e101a285 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33283e314eb7455baa5f4d41e101a285; no voters:
I20260928 06:39:43.273715 26881 leader_election.cc:288] T 00000000000000000000000000000000 P 33283e314eb7455baa5f4d41e101a285 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:43.274055 26891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33283e314eb7455baa5f4d41e101a285 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:43.274333 26891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33283e314eb7455baa5f4d41e101a285 [term 1 LEADER]: Becoming Leader. State: Replica: 33283e314eb7455baa5f4d41e101a285, State: Running, Role: LEADER
I20260928 06:39:43.274703 26884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:43.272684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:51995 (local address 127.0.0.1:32913)
0928 06:39:43.272798 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:43.272809 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:43.273069 (+ 260us) server_negotiation.cc:407] Connection header received
0928 06:39:43.273228 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:43.273231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:43.273265 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:43.273339 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:43.274192 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:43.274197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:43.274200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:43.274214 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:43.274245 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:43.274263 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:43.274357 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:43.274547 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 06:39:43.274588 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:39:43.274672 26891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33283e314eb7455baa5f4d41e101a285 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33283e314eb7455baa5f4d41e101a285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32913 } }
I20260928 06:39:43.275053 26881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33283e314eb7455baa5f4d41e101a285 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:43.276473 26893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33283e314eb7455baa5f4d41e101a285 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33283e314eb7455baa5f4d41e101a285. Latest consensus state: current_term: 1 leader_uuid: "33283e314eb7455baa5f4d41e101a285" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33283e314eb7455baa5f4d41e101a285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32913 } } }
I20260928 06:39:43.276600 26893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33283e314eb7455baa5f4d41e101a285 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:43.276973 26892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33283e314eb7455baa5f4d41e101a285 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33283e314eb7455baa5f4d41e101a285" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33283e314eb7455baa5f4d41e101a285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32913 } } }
I20260928 06:39:43.277029 26901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:43.277079 26892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33283e314eb7455baa5f4d41e101a285 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:43.280335 26901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:43.285863 26901 catalog_manager.cc:1347] Generated new cluster ID: 2f5b11c9af8444329007094b578c4b07
I20260928 06:39:43.285960 26901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:43.305758 26901 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:43.306131 26901 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:43.311933 26901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33283e314eb7455baa5f4d41e101a285: Generated new TSK 0
I20260928 06:39:43.312628 26901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:43.323827 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 26812
2026-09-28T06:39:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:43 dist-test-slave-j0qf krb5kdc[26912](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:43 dist-test-slave-j0qf krb5kdc[26912](info): set up 2 sockets
Sep 28 06:39:43 dist-test-slave-j0qf krb5kdc[26912](info): commencing operation
krb5kdc: starting...
W20260928 06:39:44.357014 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:39:44 dist-test-slave-j0qf krb5kdc[26912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:39:44.481055 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:44195
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:39499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:44195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:44.590716 26925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:44.591001 26925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:44.591071 26925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:44.595042 26925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:44.595121 26925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:44.595196 26925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:44.595249 26925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:44.595286 26925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:44.600770 26925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:39499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:44195
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:44195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:44.601804 26925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:44.602864 26925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:44.611166 26930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:44.611282 26925 server_base.cc:1034] running on GCE node
W20260928 06:39:44.611166 26933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:44.611191 26931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:44.611948 26925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:44.613071 26925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:44.614277 26925 hybrid_clock.cc:648] HybridClock initialized: now 1790577584614265 us; error 32 us; skew 500 ppm
Sep 28 06:39:44 dist-test-slave-j0qf krb5kdc[26912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577584, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:44.616991 26925 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:39:44.617934 26925 webserver.cc:460] Webserver started at http://127.5.22.62:35407/ using document root <none> and password file <none>
I20260928 06:39:44.618549 26925 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:44.618638 26925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:44.618862 26925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:44.620869 26925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "239598321a8e48289b11c8fcd2730f69"
format_stamp: "Formatted at 2026-09-28 06:39:44 on dist-test-slave-j0qf"
server_key: "5593a90831feeea556e086dd67a60021"
server_key_iv: "fc73c8bbae5090a034f60fec86d556ba"
server_key_version: "encryptionkey@0"
I20260928 06:39:44.621464 26925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "239598321a8e48289b11c8fcd2730f69"
format_stamp: "Formatted at 2026-09-28 06:39:44 on dist-test-slave-j0qf"
server_key: "5593a90831feeea556e086dd67a60021"
server_key_iv: "fc73c8bbae5090a034f60fec86d556ba"
server_key_version: "encryptionkey@0"
I20260928 06:39:44.625341 26925 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 06:39:44.627939 26940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:44.629217 26925 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:39:44.629375 26925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "239598321a8e48289b11c8fcd2730f69"
format_stamp: "Formatted at 2026-09-28 06:39:44 on dist-test-slave-j0qf"
server_key: "5593a90831feeea556e086dd67a60021"
server_key_iv: "fc73c8bbae5090a034f60fec86d556ba"
server_key_version: "encryptionkey@0"
I20260928 06:39:44.629500 26925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:44.648465 26925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:44.651847 26925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:44.652068 26925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:44.659735 26925 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:44195
I20260928 06:39:44.659752 26993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:44195 every 8 connection(s)
I20260928 06:39:44.660699 26925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:44.663396 26994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:44.667227 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 26925
I20260928 06:39:44.667383 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:44.667666 5208 external_mini_cluster.cc:1428] Setting key 7fb983221bd4c48f7ccaacf74d8c2a0b
I20260928 06:39:44.669364 26994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 239598321a8e48289b11c8fcd2730f69: Bootstrap starting.
I20260928 06:39:44.671870 26994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 239598321a8e48289b11c8fcd2730f69: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:44.672601 26994 log.cc:824] T 00000000000000000000000000000000 P 239598321a8e48289b11c8fcd2730f69: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:44.674444 26994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 239598321a8e48289b11c8fcd2730f69: No bootstrap required, opened a new log
Sep 28 06:39:44 dist-test-slave-j0qf krb5kdc[26912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:39:44.677861 26994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 239598321a8e48289b11c8fcd2730f69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "239598321a8e48289b11c8fcd2730f69" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 44195 } }
I20260928 06:39:44.678097 26994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 239598321a8e48289b11c8fcd2730f69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:44.678175 26994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 239598321a8e48289b11c8fcd2730f69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 239598321a8e48289b11c8fcd2730f69, State: Initialized, Role: FOLLOWER
I20260928 06:39:44.678687 26994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 239598321a8e48289b11c8fcd2730f69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "239598321a8e48289b11c8fcd2730f69" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 44195 } }
I20260928 06:39:44.678850 26994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 239598321a8e48289b11c8fcd2730f69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:44.678941 26994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 239598321a8e48289b11c8fcd2730f69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:44.679037 26994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 239598321a8e48289b11c8fcd2730f69 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:44.679893 26994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 239598321a8e48289b11c8fcd2730f69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "239598321a8e48289b11c8fcd2730f69" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 44195 } }
I20260928 06:39:44.680256 26994 leader_election.cc:302] T 00000000000000000000000000000000 P 239598321a8e48289b11c8fcd2730f69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 239598321a8e48289b11c8fcd2730f69; no voters:
I20260928 06:39:44.680632 26994 leader_election.cc:288] T 00000000000000000000000000000000 P 239598321a8e48289b11c8fcd2730f69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:44.680720 26999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 239598321a8e48289b11c8fcd2730f69 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:44.680903 26999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 239598321a8e48289b11c8fcd2730f69 [term 1 LEADER]: Becoming Leader. State: Replica: 239598321a8e48289b11c8fcd2730f69, State: Running, Role: LEADER
I20260928 06:39:44.681330 26999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 239598321a8e48289b11c8fcd2730f69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "239598321a8e48289b11c8fcd2730f69" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 44195 } }
I20260928 06:39:44.681906 26994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 239598321a8e48289b11c8fcd2730f69 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:44.682471 26999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 239598321a8e48289b11c8fcd2730f69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 239598321a8e48289b11c8fcd2730f69. Latest consensus state: current_term: 1 leader_uuid: "239598321a8e48289b11c8fcd2730f69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "239598321a8e48289b11c8fcd2730f69" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 44195 } } }
I20260928 06:39:44.682595 26999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 239598321a8e48289b11c8fcd2730f69 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:44.683116 26997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:44.669058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44057 (local address 127.5.22.62:44195)
0928 06:39:44.669487 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:44.669507 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:44.669558 (+ 51us) server_negotiation.cc:407] Connection header received
0928 06:39:44.670247 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:44.670265 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:44.670693 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:44.670967 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:44.671876 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:44.673003 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:44.673927 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:44.674472 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:44.674654 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:44.677194 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:44.677199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:44.677212 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:44.677239 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:44.679322 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:44.679832 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:44.679837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:44.679843 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:44.679961 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:44.680429 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:44.680434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:44.680437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:44.680953 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:44.681715 (+ 762us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:44.681855 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 06:39:44.682102 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":155,"threads_started":1}
I20260928 06:39:44.683295 27000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 239598321a8e48289b11c8fcd2730f69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "239598321a8e48289b11c8fcd2730f69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "239598321a8e48289b11c8fcd2730f69" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 44195 } } }
I20260928 06:39:44.683420 27000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 239598321a8e48289b11c8fcd2730f69 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:44.684249 27003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:44.687261 27003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:44.687855 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:44.693351 27003 catalog_manager.cc:1347] Generated new cluster ID: 49f82f996e13446392563142c9d4401f
I20260928 06:39:44.693423 27003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:44.702059 26997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:44.690099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39949 (local address 127.5.22.62:44195)
0928 06:39:44.690214 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:44.690218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:44.690484 (+ 266us) server_negotiation.cc:407] Connection header received
0928 06:39:44.690604 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:44.690607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:44.690654 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:44.690746 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:44.691482 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:44.692310 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:44.693515 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:44.693917 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:44.694020 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:44.698550 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:44.698552 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:44.698554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:44.698567 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:44.700129 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:44.700632 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:44.700637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:44.700639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:44.700723 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:44.701062 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:44.701065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:44.701067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:44.701250 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:44.701403 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:44.701675 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 06:39:44.701770 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":43}
I20260928 06:39:44.709887 27003 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:44.710904 27003 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:44.723577 27003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 239598321a8e48289b11c8fcd2730f69: Generated new TSK 0
I20260928 06:39:44.724530 27003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:44.730808 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 26925
W20260928 06:39:44.736413 26910 connection.cc:535] client connection to 127.5.22.62:44195 recv error: Network error: failed to read from TLS socket (remote: 127.5.22.62:44195): Connection reset by peer (error 104)
2026-09-28T06:39:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:44 dist-test-slave-j0qf krb5kdc[27025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:44 dist-test-slave-j0qf krb5kdc[27025](info): set up 2 sockets
Sep 28 06:39:44 dist-test-slave-j0qf krb5kdc[27025](info): commencing operation
krb5kdc: starting...
W20260928 06:39:45.777140 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:39:45 dist-test-slave-j0qf krb5kdc[27025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:39:45.901463 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:42613
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:40781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:42613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:46.030676 27038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:46.031123 27038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:46.031236 27038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:46.036936 27038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:46.037012 27038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:46.037032 27038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:46.037050 27038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:46.037065 27038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:46.042697 27038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:40781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:42613
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:42613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:46.043746 27038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:46.044749 27038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:46.053792 27046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:46.053998 27038 server_base.cc:1034] running on GCE node
W20260928 06:39:46.053781 27044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:46.053781 27043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:46.054965 27038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:46.056186 27038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:46.057372 27038 hybrid_clock.cc:648] HybridClock initialized: now 1790577586057353 us; error 35 us; skew 500 ppm
Sep 28 06:39:46 dist-test-slave-j0qf krb5kdc[27025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577586, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:46.060161 27038 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:39:46.061084 27038 webserver.cc:460] Webserver started at http://127.5.22.62:40029/ using document root <none> and password file <none>
I20260928 06:39:46.061597 27038 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:46.061654 27038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:46.061816 27038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:46.063831 27038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "021c487245374098af599e012a79e3e9"
format_stamp: "Formatted at 2026-09-28 06:39:46 on dist-test-slave-j0qf"
server_key: "67b4e1ac695b0d893d3ba3badb3d8951"
server_key_iv: "71487c1390509cd7af89c6fb2758cfbd"
server_key_version: "encryptionkey@0"
I20260928 06:39:46.064400 27038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "021c487245374098af599e012a79e3e9"
format_stamp: "Formatted at 2026-09-28 06:39:46 on dist-test-slave-j0qf"
server_key: "67b4e1ac695b0d893d3ba3badb3d8951"
server_key_iv: "71487c1390509cd7af89c6fb2758cfbd"
server_key_version: "encryptionkey@0"
I20260928 06:39:46.068034 27038 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260928 06:39:46.070559 27053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:46.071764 27038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:39:46.071913 27038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "021c487245374098af599e012a79e3e9"
format_stamp: "Formatted at 2026-09-28 06:39:46 on dist-test-slave-j0qf"
server_key: "67b4e1ac695b0d893d3ba3badb3d8951"
server_key_iv: "71487c1390509cd7af89c6fb2758cfbd"
server_key_version: "encryptionkey@0"
I20260928 06:39:46.072024 27038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:46.086253 27038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:46.089370 27038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:46.089576 27038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:46.097201 27038 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:42613
I20260928 06:39:46.097244 27105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:42613 every 8 connection(s)
I20260928 06:39:46.098187 27038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:46.101115 27106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:46.102254 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 27038
I20260928 06:39:46.102427 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:46.102774 5208 external_mini_cluster.cc:1428] Setting key 4d9ecb86437127a317118990f117a37b
I20260928 06:39:46.107518 27106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 021c487245374098af599e012a79e3e9: Bootstrap starting.
I20260928 06:39:46.109740 27106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 021c487245374098af599e012a79e3e9: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:46.110546 27106 log.cc:824] T 00000000000000000000000000000000 P 021c487245374098af599e012a79e3e9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:46.111794 27109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:46.104194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39623 (local address 127.5.22.62:42613)
0928 06:39:46.104633 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:46.104647 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:46.104681 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:39:46.105338 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:46.105356 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:46.105690 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:46.106014 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:46.106492 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:46.107501 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:46.108443 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:46.108913 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:46.109076 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:46.110046 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:46.110051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:46.110063 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:46.110524 (+ 461us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:46.110578 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:46.110599 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:46.110745 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:46.110948 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 06:39:46.111085 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":138,"threads_started":1}
I20260928 06:39:46.112720 27106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 021c487245374098af599e012a79e3e9: No bootstrap required, opened a new log
I20260928 06:39:46.114845 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:46.116647 27106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 021c487245374098af599e012a79e3e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021c487245374098af599e012a79e3e9" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 42613 } }
I20260928 06:39:46.117033 27106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 021c487245374098af599e012a79e3e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:46.117130 27106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 021c487245374098af599e012a79e3e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021c487245374098af599e012a79e3e9, State: Initialized, Role: FOLLOWER
I20260928 06:39:46.117636 27106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 021c487245374098af599e012a79e3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021c487245374098af599e012a79e3e9" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 42613 } }
I20260928 06:39:46.117830 27106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 021c487245374098af599e012a79e3e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:46.117906 27106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 021c487245374098af599e012a79e3e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:46.118016 27106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 021c487245374098af599e012a79e3e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:46.119194 27106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 021c487245374098af599e012a79e3e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021c487245374098af599e012a79e3e9" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 42613 } }
I20260928 06:39:46.119555 27106 leader_election.cc:302] T 00000000000000000000000000000000 P 021c487245374098af599e012a79e3e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 021c487245374098af599e012a79e3e9; no voters:
I20260928 06:39:46.119899 27106 leader_election.cc:288] T 00000000000000000000000000000000 P 021c487245374098af599e012a79e3e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:46.120008 27116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 021c487245374098af599e012a79e3e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:46.120239 27116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 021c487245374098af599e012a79e3e9 [term 1 LEADER]: Becoming Leader. State: Replica: 021c487245374098af599e012a79e3e9, State: Running, Role: LEADER
I20260928 06:39:46.120656 27116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 021c487245374098af599e012a79e3e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021c487245374098af599e012a79e3e9" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 42613 } }
I20260928 06:39:46.121225 27106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 021c487245374098af599e012a79e3e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:46.121899 27109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:46.116875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44055 (local address 127.5.22.62:42613)
0928 06:39:46.116987 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:46.116991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:46.117151 (+ 160us) server_negotiation.cc:407] Connection header received
0928 06:39:46.117282 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:46.117285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:46.117330 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:46.117461 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:46.117834 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:46.118553 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:46.119727 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:46.120142 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:46.120254 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:46.121280 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:46.121283 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:46.121286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:46.121298 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:46.121326 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:46.121339 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:46.121413 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:46.121728 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 06:39:46.121761 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:39:46.122860 27117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 021c487245374098af599e012a79e3e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "021c487245374098af599e012a79e3e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021c487245374098af599e012a79e3e9" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 42613 } } }
I20260928 06:39:46.122996 27117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 021c487245374098af599e012a79e3e9 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:46.123054 27118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 021c487245374098af599e012a79e3e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 021c487245374098af599e012a79e3e9. Latest consensus state: current_term: 1 leader_uuid: "021c487245374098af599e012a79e3e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021c487245374098af599e012a79e3e9" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 42613 } } }
I20260928 06:39:46.123127 27118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 021c487245374098af599e012a79e3e9 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:46.123422 27129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:46.126468 27129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:46.131790 27129 catalog_manager.cc:1347] Generated new cluster ID: b2a5ae36037b4c04baf5f6cccc91370d
I20260928 06:39:46.131862 27129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:46.143064 27129 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:46.143941 27129 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:46.152185 27129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 021c487245374098af599e012a79e3e9: Generated new TSK 0
I20260928 06:39:46.152916 27129 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:46.162513 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 27038
W20260928 06:39:46.168756 27023 connection.cc:535] client connection to 127.5.22.62:42613 recv error: Network error: failed to read from TLS socket (remote: 127.5.22.62:42613): Connection reset by peer (error 104)
2026-09-28T06:39:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:46 dist-test-slave-j0qf krb5kdc[27138](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:46 dist-test-slave-j0qf krb5kdc[27138](info): set up 2 sockets
Sep 28 06:39:46 dist-test-slave-j0qf krb5kdc[27138](info): commencing operation
krb5kdc: starting...
W20260928 06:39:47.262698 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:39:47 dist-test-slave-j0qf krb5kdc[27138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:39:47.386690 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:35661
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:38651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:35661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:47.499142 27151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:47.499408 27151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:47.499450 27151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:47.503688 27151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:47.503746 27151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:47.503767 27151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:47.503782 27151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:47.503836 27151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:47.509294 27151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:38651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:35661
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:35661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:47.510418 27151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:47.511538 27151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:47.520310 27157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:47.520323 27159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:47.520303 27156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:47.520498 27151 server_base.cc:1034] running on GCE node
I20260928 06:39:47.521162 27151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:47.522239 27151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:47.523404 27151 hybrid_clock.cc:648] HybridClock initialized: now 1790577587523393 us; error 29 us; skew 500 ppm
Sep 28 06:39:47 dist-test-slave-j0qf krb5kdc[27138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577587, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:47.526170 27151 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:39:47.527129 27151 webserver.cc:460] Webserver started at http://127.5.22.62:34561/ using document root <none> and password file <none>
I20260928 06:39:47.527701 27151 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:47.527767 27151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:47.527940 27151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:47.530014 27151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "009382865ecd4e85a19b7660ec7b538c"
format_stamp: "Formatted at 2026-09-28 06:39:47 on dist-test-slave-j0qf"
server_key: "0262e2b585f7b1248bbf7923f3892c1d"
server_key_iv: "65d8309651a5392113360fed8221c6f8"
server_key_version: "encryptionkey@0"
I20260928 06:39:47.530614 27151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "009382865ecd4e85a19b7660ec7b538c"
format_stamp: "Formatted at 2026-09-28 06:39:47 on dist-test-slave-j0qf"
server_key: "0262e2b585f7b1248bbf7923f3892c1d"
server_key_iv: "65d8309651a5392113360fed8221c6f8"
server_key_version: "encryptionkey@0"
I20260928 06:39:47.534543 27151 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 06:39:47.536944 27166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:47.538254 27151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:39:47.538453 27151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "009382865ecd4e85a19b7660ec7b538c"
format_stamp: "Formatted at 2026-09-28 06:39:47 on dist-test-slave-j0qf"
server_key: "0262e2b585f7b1248bbf7923f3892c1d"
server_key_iv: "65d8309651a5392113360fed8221c6f8"
server_key_version: "encryptionkey@0"
I20260928 06:39:47.538556 27151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:47.543219 27151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:47.546303 27151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:47.546617 27151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:47.554534 27151 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:35661
I20260928 06:39:47.554615 27218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:35661 every 8 connection(s)
I20260928 06:39:47.555506 27151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:47.558189 27219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:47.562855 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 27151
I20260928 06:39:47.563022 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:47.563301 5208 external_mini_cluster.cc:1428] Setting key 2848c89fafdd9b0ea1955309d9a30637
I20260928 06:39:47.564301 27219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 009382865ecd4e85a19b7660ec7b538c: Bootstrap starting.
I20260928 06:39:47.566604 27219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 009382865ecd4e85a19b7660ec7b538c: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:47.567625 27219 log.cc:824] T 00000000000000000000000000000000 P 009382865ecd4e85a19b7660ec7b538c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:47.568496 27222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:47.564691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37129 (local address 127.5.22.62:35661)
0928 06:39:47.565101 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:47.565122 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:47.565169 (+ 47us) server_negotiation.cc:407] Connection header received
0928 06:39:47.565806 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:47.565823 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:47.566100 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:47.566450 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:47.567235 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:47.567240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:47.567251 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:47.567261 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:47.567315 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:47.567335 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:47.567425 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:47.567578 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 06:39:47.567771 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":132,"threads_started":1}
I20260928 06:39:47.569747 27219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 009382865ecd4e85a19b7660ec7b538c: No bootstrap required, opened a new log
I20260928 06:39:47.571480 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:47.573712 27219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 009382865ecd4e85a19b7660ec7b538c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009382865ecd4e85a19b7660ec7b538c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35661 } }
I20260928 06:39:47.574010 27219 raft_consensus.cc:374] T 00000000000000000000000000000000 P 009382865ecd4e85a19b7660ec7b538c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:47.574131 27219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 009382865ecd4e85a19b7660ec7b538c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009382865ecd4e85a19b7660ec7b538c, State: Initialized, Role: FOLLOWER
I20260928 06:39:47.574620 27219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 009382865ecd4e85a19b7660ec7b538c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009382865ecd4e85a19b7660ec7b538c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35661 } }
I20260928 06:39:47.574746 27219 raft_consensus.cc:388] T 00000000000000000000000000000000 P 009382865ecd4e85a19b7660ec7b538c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:47.574826 27219 raft_consensus.cc:482] T 00000000000000000000000000000000 P 009382865ecd4e85a19b7660ec7b538c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:47.574949 27219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 009382865ecd4e85a19b7660ec7b538c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:47.575321 27222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:47.573555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48911 (local address 127.5.22.62:35661)
0928 06:39:47.573680 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:47.573685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:47.573840 (+ 155us) server_negotiation.cc:407] Connection header received
0928 06:39:47.573910 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:47.573913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:47.573948 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:47.574036 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:47.574903 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:47.574907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:47.574909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:47.574920 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:47.574946 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:47.574965 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:47.575057 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:47.575134 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 06:39:47.575193 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 06:39:47.576167 27219 raft_consensus.cc:504] T 00000000000000000000000000000000 P 009382865ecd4e85a19b7660ec7b538c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009382865ecd4e85a19b7660ec7b538c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35661 } }
I20260928 06:39:47.576525 27219 leader_election.cc:302] T 00000000000000000000000000000000 P 009382865ecd4e85a19b7660ec7b538c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 009382865ecd4e85a19b7660ec7b538c; no voters:
I20260928 06:39:47.576835 27219 leader_election.cc:288] T 00000000000000000000000000000000 P 009382865ecd4e85a19b7660ec7b538c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:47.576915 27229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 009382865ecd4e85a19b7660ec7b538c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:47.577164 27229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 009382865ecd4e85a19b7660ec7b538c [term 1 LEADER]: Becoming Leader. State: Replica: 009382865ecd4e85a19b7660ec7b538c, State: Running, Role: LEADER
I20260928 06:39:47.577508 27229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 009382865ecd4e85a19b7660ec7b538c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009382865ecd4e85a19b7660ec7b538c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35661 } }
I20260928 06:39:47.578055 27219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 009382865ecd4e85a19b7660ec7b538c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:47.579263 27230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 009382865ecd4e85a19b7660ec7b538c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "009382865ecd4e85a19b7660ec7b538c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009382865ecd4e85a19b7660ec7b538c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35661 } } }
I20260928 06:39:47.579419 27230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 009382865ecd4e85a19b7660ec7b538c [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:47.579255 27231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 009382865ecd4e85a19b7660ec7b538c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 009382865ecd4e85a19b7660ec7b538c. Latest consensus state: current_term: 1 leader_uuid: "009382865ecd4e85a19b7660ec7b538c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009382865ecd4e85a19b7660ec7b538c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35661 } } }
I20260928 06:39:47.579735 27231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 009382865ecd4e85a19b7660ec7b538c [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:47.579830 27238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:47.582820 27238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:47.588479 27238 catalog_manager.cc:1347] Generated new cluster ID: b59c462eaceb4315b881924e96e2bf92
I20260928 06:39:47.588544 27238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:47.613255 27238 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:47.613644 27238 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:47.627555 27238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 009382865ecd4e85a19b7660ec7b538c: Generated new TSK 0
I20260928 06:39:47.628513 27238 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:47.639780 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 27151
2026-09-28T06:39:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1476 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:47 dist-test-slave-j0qf krb5kdc[27250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:47 dist-test-slave-j0qf krb5kdc[27250](info): set up 2 sockets
Sep 28 06:39:47 dist-test-slave-j0qf krb5kdc[27250](info): commencing operation
krb5kdc: starting...
W20260928 06:39:48.676994 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:39:48 dist-test-slave-j0qf krb5kdc[27250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:39:48.803149 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:42897
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:42897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:48.915235 27263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:48.915522 27263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:48.915603 27263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:48.919837 27263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:48.919904 27263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:48.919925 27263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:48.919963 27263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:48.919997 27263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:48.925354 27263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:42897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:42897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:48.926530 27263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:48.927947 27263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:48.937414 27272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:48.937462 27270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:48.937453 27263 server_base.cc:1034] running on GCE node
W20260928 06:39:48.937417 27269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:48.938228 27263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:48.939385 27263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:48.940552 27263 hybrid_clock.cc:648] HybridClock initialized: now 1790577588940540 us; error 29 us; skew 500 ppm
Sep 28 06:39:48 dist-test-slave-j0qf krb5kdc[27250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577588, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:48.943527 27263 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:39:48.944458 27263 webserver.cc:460] Webserver started at http://127.5.22.62:37733/ using document root <none> and password file <none>
I20260928 06:39:48.945065 27263 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:48.945137 27263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:48.945323 27263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:48.947702 27263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "ab578e6604f144aca0f89c617b873139"
format_stamp: "Formatted at 2026-09-28 06:39:48 on dist-test-slave-j0qf"
server_key: "594e5b71ee546b94e321870002ecdc3a"
server_key_iv: "05ad4875db081aac79284bdb9f919c65"
server_key_version: "encryptionkey@0"
I20260928 06:39:48.948410 27263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "ab578e6604f144aca0f89c617b873139"
format_stamp: "Formatted at 2026-09-28 06:39:48 on dist-test-slave-j0qf"
server_key: "594e5b71ee546b94e321870002ecdc3a"
server_key_iv: "05ad4875db081aac79284bdb9f919c65"
server_key_version: "encryptionkey@0"
I20260928 06:39:48.952365 27263 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:39:48.954785 27279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:48.956077 27263 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:39:48.956278 27263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "ab578e6604f144aca0f89c617b873139"
format_stamp: "Formatted at 2026-09-28 06:39:48 on dist-test-slave-j0qf"
server_key: "594e5b71ee546b94e321870002ecdc3a"
server_key_iv: "05ad4875db081aac79284bdb9f919c65"
server_key_version: "encryptionkey@0"
I20260928 06:39:48.956382 27263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:48.981384 27263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:48.984786 27263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:48.985000 27263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:48.992813 27263 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:42897
I20260928 06:39:48.992868 27331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:42897 every 8 connection(s)
I20260928 06:39:48.993820 27263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:48.996721 27332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:49.000348 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 27263
I20260928 06:39:49.000525 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:49.000792 5208 external_mini_cluster.cc:1428] Setting key 7364715bc47e41bec90bad2a28c6f610
I20260928 06:39:49.002631 27332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab578e6604f144aca0f89c617b873139: Bootstrap starting.
I20260928 06:39:49.004798 27332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab578e6604f144aca0f89c617b873139: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:49.005529 27332 log.cc:824] T 00000000000000000000000000000000 P ab578e6604f144aca0f89c617b873139: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:49.007383 27332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab578e6604f144aca0f89c617b873139: No bootstrap required, opened a new log
Sep 28 06:39:49 dist-test-slave-j0qf krb5kdc[27250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:39:49.010908 27332 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab578e6604f144aca0f89c617b873139 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab578e6604f144aca0f89c617b873139" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 42897 } }
I20260928 06:39:49.011195 27332 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab578e6604f144aca0f89c617b873139 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:49.011281 27332 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab578e6604f144aca0f89c617b873139 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab578e6604f144aca0f89c617b873139, State: Initialized, Role: FOLLOWER
I20260928 06:39:49.011752 27332 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab578e6604f144aca0f89c617b873139 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab578e6604f144aca0f89c617b873139" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 42897 } }
I20260928 06:39:49.011900 27332 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab578e6604f144aca0f89c617b873139 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:49.012008 27332 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab578e6604f144aca0f89c617b873139 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:49.012145 27332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab578e6604f144aca0f89c617b873139 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:49.013152 27332 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab578e6604f144aca0f89c617b873139 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab578e6604f144aca0f89c617b873139" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 42897 } }
I20260928 06:39:49.013527 27332 leader_election.cc:302] T 00000000000000000000000000000000 P ab578e6604f144aca0f89c617b873139 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab578e6604f144aca0f89c617b873139; no voters:
I20260928 06:39:49.013867 27332 leader_election.cc:288] T 00000000000000000000000000000000 P ab578e6604f144aca0f89c617b873139 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:49.013927 27337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab578e6604f144aca0f89c617b873139 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:49.014168 27337 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab578e6604f144aca0f89c617b873139 [term 1 LEADER]: Becoming Leader. State: Replica: ab578e6604f144aca0f89c617b873139, State: Running, Role: LEADER
I20260928 06:39:49.014555 27337 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab578e6604f144aca0f89c617b873139 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab578e6604f144aca0f89c617b873139" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 42897 } }
I20260928 06:39:49.014933 27335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:49.002210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51827 (local address 127.5.22.62:42897)
0928 06:39:49.002686 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:49.002701 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:49.002739 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:39:49.003413 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:49.003429 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:49.003778 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:49.004110 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:49.004916 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:49.005979 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:49.006911 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:49.007373 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:49.007535 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:49.009988 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:49.009993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:49.010005 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:49.010030 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:49.011852 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:49.012387 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:49.012391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:49.012396 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:49.012500 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:49.012795 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:49.012797 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:49.012798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:49.013150 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:49.013347 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:49.013581 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 06:39:49.013915 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":133,"threads_started":1}
I20260928 06:39:49.015136 27332 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab578e6604f144aca0f89c617b873139 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:49.015682 27337 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab578e6604f144aca0f89c617b873139 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab578e6604f144aca0f89c617b873139. Latest consensus state: current_term: 1 leader_uuid: "ab578e6604f144aca0f89c617b873139" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab578e6604f144aca0f89c617b873139" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 42897 } } }
I20260928 06:39:49.015884 27337 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab578e6604f144aca0f89c617b873139 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:49.016090 27338 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab578e6604f144aca0f89c617b873139 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab578e6604f144aca0f89c617b873139" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab578e6604f144aca0f89c617b873139" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 42897 } } }
I20260928 06:39:49.016279 27338 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab578e6604f144aca0f89c617b873139 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:49.016332 27344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:49.018112 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:49.019541 27344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:49.030180 27344 catalog_manager.cc:1347] Generated new cluster ID: fd4e61de2efb4fc7a3f4a7d186b10741
I20260928 06:39:49.030249 27344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:49.033820 27335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:49.024483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40141 (local address 127.5.22.62:42897)
0928 06:39:49.024593 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:49.024598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:49.024613 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:39:49.024651 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:49.024655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:49.024711 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:49.024801 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:49.025797 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:49.027290 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:49.028264 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:49.028610 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:49.028724 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:49.030378 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:49.030382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:49.030385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:49.030408 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:49.032127 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:49.032609 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:49.032613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:49.032614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:49.032701 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:49.032992 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:49.032995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:49.032997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:49.033174 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:49.033280 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:49.033559 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 06:39:49.033664 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:39:49.042398 27344 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:49.043375 27344 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:49.057368 27344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab578e6604f144aca0f89c617b873139: Generated new TSK 0
I20260928 06:39:49.058107 27344 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:49.067943 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 27263
2026-09-28T06:39:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1428 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:49 dist-test-slave-j0qf krb5kdc[27363](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:49 dist-test-slave-j0qf krb5kdc[27363](info): set up 2 sockets
Sep 28 06:39:49 dist-test-slave-j0qf krb5kdc[27363](info): commencing operation
krb5kdc: starting...
W20260928 06:39:50.104904 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:39:50 dist-test-slave-j0qf krb5kdc[27363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:39:50.229192 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:40405
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:42407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:40405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:50.340387 27376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:50.340657 27376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:50.340699 27376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:50.344868 27376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:50.344933 27376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:50.344954 27376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:50.344970 27376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:50.345032 27376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:50.350349 27376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:42407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:40405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:40405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:50.351409 27376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:50.352435 27376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:50.360998 27382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:50.361014 27384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:50.361106 27376 server_base.cc:1034] running on GCE node
W20260928 06:39:50.361084 27381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:50.361860 27376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:50.363051 27376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:50.364259 27376 hybrid_clock.cc:648] HybridClock initialized: now 1790577590364226 us; error 43 us; skew 500 ppm
Sep 28 06:39:50 dist-test-slave-j0qf krb5kdc[27363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577590, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:50.367005 27376 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:39:50.368220 27376 webserver.cc:460] Webserver started at http://127.5.22.62:35603/ using document root <none> and password file <none>
I20260928 06:39:50.368777 27376 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:50.368837 27376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:50.369042 27376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:50.371119 27376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "9d52162488004209acc9f3bf517d5c18"
format_stamp: "Formatted at 2026-09-28 06:39:50 on dist-test-slave-j0qf"
server_key: "4cb25289ebb299860f3fbb27905ae34b"
server_key_iv: "3286720d68d14de79cd6433822565e46"
server_key_version: "encryptionkey@0"
I20260928 06:39:50.371670 27376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "9d52162488004209acc9f3bf517d5c18"
format_stamp: "Formatted at 2026-09-28 06:39:50 on dist-test-slave-j0qf"
server_key: "4cb25289ebb299860f3fbb27905ae34b"
server_key_iv: "3286720d68d14de79cd6433822565e46"
server_key_version: "encryptionkey@0"
I20260928 06:39:50.375481 27376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 06:39:50.377916 27391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:50.379305 27376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 06:39:50.379460 27376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "9d52162488004209acc9f3bf517d5c18"
format_stamp: "Formatted at 2026-09-28 06:39:50 on dist-test-slave-j0qf"
server_key: "4cb25289ebb299860f3fbb27905ae34b"
server_key_iv: "3286720d68d14de79cd6433822565e46"
server_key_version: "encryptionkey@0"
I20260928 06:39:50.379575 27376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:50.424006 27376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:50.427557 27376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:50.427780 27376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:50.435006 27376 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:40405
I20260928 06:39:50.435017 27443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:40405 every 8 connection(s)
I20260928 06:39:50.435992 27376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:50.438817 27444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:50.444291 27444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d52162488004209acc9f3bf517d5c18: Bootstrap starting.
I20260928 06:39:50.445953 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 27376
I20260928 06:39:50.446095 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:50.446425 5208 external_mini_cluster.cc:1428] Setting key 669878a3c198b3ac2515910dba70c961
I20260928 06:39:50.446998 27444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d52162488004209acc9f3bf517d5c18: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:50.447937 27444 log.cc:824] T 00000000000000000000000000000000 P 9d52162488004209acc9f3bf517d5c18: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:50.449975 27444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d52162488004209acc9f3bf517d5c18: No bootstrap required, opened a new log
I20260928 06:39:50.454146 27444 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d52162488004209acc9f3bf517d5c18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d52162488004209acc9f3bf517d5c18" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 40405 } }
I20260928 06:39:50.454445 27444 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d52162488004209acc9f3bf517d5c18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:50.454551 27444 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d52162488004209acc9f3bf517d5c18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d52162488004209acc9f3bf517d5c18, State: Initialized, Role: FOLLOWER
I20260928 06:39:50.455034 27444 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d52162488004209acc9f3bf517d5c18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d52162488004209acc9f3bf517d5c18" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 40405 } }
I20260928 06:39:50.455184 27444 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d52162488004209acc9f3bf517d5c18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:50.455268 27444 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d52162488004209acc9f3bf517d5c18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:50.455395 27444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d52162488004209acc9f3bf517d5c18 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:50.455713 27448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:50.447599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49735 (local address 127.5.22.62:40405)
0928 06:39:50.448022 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:50.448044 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:50.448085 (+ 41us) server_negotiation.cc:407] Connection header received
0928 06:39:50.448930 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:50.448949 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:50.449272 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:50.449547 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:50.449912 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:50.451023 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:50.452255 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:50.452717 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:50.452857 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:50.453694 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:50.453699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:50.453711 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:50.454120 (+ 409us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:50.454176 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:50.454198 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:50.454275 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:50.454489 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 06:39:50.454726 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":140,"threads_started":1}
I20260928 06:39:50.456362 27444 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d52162488004209acc9f3bf517d5c18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d52162488004209acc9f3bf517d5c18" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 40405 } }
I20260928 06:39:50.456733 27444 leader_election.cc:302] T 00000000000000000000000000000000 P 9d52162488004209acc9f3bf517d5c18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d52162488004209acc9f3bf517d5c18; no voters:
I20260928 06:39:50.457139 27444 leader_election.cc:288] T 00000000000000000000000000000000 P 9d52162488004209acc9f3bf517d5c18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:50.457250 27450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d52162488004209acc9f3bf517d5c18 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:50.457551 27450 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d52162488004209acc9f3bf517d5c18 [term 1 LEADER]: Becoming Leader. State: Replica: 9d52162488004209acc9f3bf517d5c18, State: Running, Role: LEADER
I20260928 06:39:50.457940 27450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d52162488004209acc9f3bf517d5c18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d52162488004209acc9f3bf517d5c18" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 40405 } }
I20260928 06:39:50.458523 27444 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d52162488004209acc9f3bf517d5c18 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:50.459053 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:50.461513 27451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d52162488004209acc9f3bf517d5c18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d52162488004209acc9f3bf517d5c18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d52162488004209acc9f3bf517d5c18" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 40405 } } }
I20260928 06:39:50.461750 27451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d52162488004209acc9f3bf517d5c18 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:50.461654 27452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d52162488004209acc9f3bf517d5c18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d52162488004209acc9f3bf517d5c18. Latest consensus state: current_term: 1 leader_uuid: "9d52162488004209acc9f3bf517d5c18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d52162488004209acc9f3bf517d5c18" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 40405 } } }
I20260928 06:39:50.462073 27452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d52162488004209acc9f3bf517d5c18 [sys.catalog]: This master's current role is: LEADER
W20260928 06:39:50.464146 27468 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d52162488004209acc9f3bf517d5c18: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:39:50.464233 27468 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:39:50.464332 27471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:50.465229 27471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:50.466369 27448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:50.461455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49879 (local address 127.5.22.62:40405)
0928 06:39:50.461566 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:50.461571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:50.461959 (+ 388us) server_negotiation.cc:407] Connection header received
0928 06:39:50.462143 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:50.462146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:50.462194 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:50.462246 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:50.462674 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:50.463376 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:50.464568 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:50.464911 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:50.465033 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:50.465890 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:50.465893 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:50.465896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:50.465908 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:50.465936 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:50.465949 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:50.466026 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:50.466157 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 06:39:50.466209 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:39:50.471356 27471 catalog_manager.cc:1347] Generated new cluster ID: 2d5f8487869441aba813097d8f35cefe
I20260928 06:39:50.471429 27471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:50.476097 27471 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:50.476949 27471 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:50.485622 27471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d52162488004209acc9f3bf517d5c18: Generated new TSK 0
I20260928 06:39:50.486281 27471 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:50.495414 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 27376
2026-09-28T06:39:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1427 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:50 dist-test-slave-j0qf krb5kdc[27476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:50 dist-test-slave-j0qf krb5kdc[27476](info): set up 2 sockets
Sep 28 06:39:50 dist-test-slave-j0qf krb5kdc[27476](info): commencing operation
krb5kdc: starting...
W20260928 06:39:51.537070 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:39:51 dist-test-slave-j0qf krb5kdc[27476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:39:51.665156 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:45879
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:36957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:45879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:51.776798 27489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:51.777058 27489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:51.777101 27489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:51.781132 27489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:51.781186 27489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:51.781206 27489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:51.781224 27489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:51.781239 27489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:51.786369 27489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:36957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:45879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:45879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:51.787421 27489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:51.788429 27489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:51.797183 27497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:51.797292 27495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:51.797169 27494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:51.797628 27489 server_base.cc:1034] running on GCE node
I20260928 06:39:51.798182 27489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:51.799314 27489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:51.800516 27489 hybrid_clock.cc:648] HybridClock initialized: now 1790577591800498 us; error 33 us; skew 500 ppm
Sep 28 06:39:51 dist-test-slave-j0qf krb5kdc[27476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577591, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:51.808064 27489 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:39:51.808991 27489 webserver.cc:460] Webserver started at http://127.5.22.62:34435/ using document root <none> and password file <none>
I20260928 06:39:51.809546 27489 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:51.809608 27489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:51.809779 27489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:51.811918 27489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "665d270236134191a4ebf76153e0d2c3"
format_stamp: "Formatted at 2026-09-28 06:39:51 on dist-test-slave-j0qf"
server_key: "a62470ea0aba24d578a19a88fc54caa6"
server_key_iv: "fc9adc4dadd35f0de45f94b1cbc1e4f7"
server_key_version: "encryptionkey@0"
I20260928 06:39:51.812458 27489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "665d270236134191a4ebf76153e0d2c3"
format_stamp: "Formatted at 2026-09-28 06:39:51 on dist-test-slave-j0qf"
server_key: "a62470ea0aba24d578a19a88fc54caa6"
server_key_iv: "fc9adc4dadd35f0de45f94b1cbc1e4f7"
server_key_version: "encryptionkey@0"
I20260928 06:39:51.816473 27489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:39:51.819031 27504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:51.820313 27489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 06:39:51.820421 27489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "665d270236134191a4ebf76153e0d2c3"
format_stamp: "Formatted at 2026-09-28 06:39:51 on dist-test-slave-j0qf"
server_key: "a62470ea0aba24d578a19a88fc54caa6"
server_key_iv: "fc9adc4dadd35f0de45f94b1cbc1e4f7"
server_key_version: "encryptionkey@0"
I20260928 06:39:51.820511 27489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:51.825173 27489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:51.828343 27489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:51.828500 27489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:51.835819 27489 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:45879
I20260928 06:39:51.835847 27556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:45879 every 8 connection(s)
I20260928 06:39:51.836789 27489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:51.839560 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:51.841490 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 27489
I20260928 06:39:51.841655 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:51.841943 5208 external_mini_cluster.cc:1428] Setting key 8c0e5ac020900eff528bb0a2d67ee08c
I20260928 06:39:51.846248 27557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 665d270236134191a4ebf76153e0d2c3: Bootstrap starting.
I20260928 06:39:51.847607 27560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:51.843441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40345 (local address 127.5.22.62:45879)
0928 06:39:51.843894 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:51.843908 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:51.843943 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:39:51.844589 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:51.844606 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:51.844889 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:51.845223 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:51.846057 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:51.846062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:51.846074 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:51.846084 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:51.846140 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:51.846159 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:51.846303 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:51.846414 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 06:39:51.846577 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":136,"threads_started":1}
I20260928 06:39:51.849116 27557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 665d270236134191a4ebf76153e0d2c3: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:51.850042 27557 log.cc:824] T 00000000000000000000000000000000 P 665d270236134191a4ebf76153e0d2c3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:51.850759 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:51.852147 27557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 665d270236134191a4ebf76153e0d2c3: No bootstrap required, opened a new log
I20260928 06:39:51.854573 27560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:51.852851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40173 (local address 127.5.22.62:45879)
0928 06:39:51.852982 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:51.852986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:51.853182 (+ 196us) server_negotiation.cc:407] Connection header received
0928 06:39:51.853298 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:51.853300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:51.853329 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:51.853421 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:51.854141 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:51.854143 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:51.854145 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:51.854156 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:51.854180 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:51.854190 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:51.854259 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:51.854403 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 06:39:51.854436 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:39:51.855894 27557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 665d270236134191a4ebf76153e0d2c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665d270236134191a4ebf76153e0d2c3" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 45879 } }
I20260928 06:39:51.856134 27557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 665d270236134191a4ebf76153e0d2c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:51.856208 27557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 665d270236134191a4ebf76153e0d2c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 665d270236134191a4ebf76153e0d2c3, State: Initialized, Role: FOLLOWER
I20260928 06:39:51.856644 27557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 665d270236134191a4ebf76153e0d2c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665d270236134191a4ebf76153e0d2c3" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 45879 } }
I20260928 06:39:51.856793 27557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 665d270236134191a4ebf76153e0d2c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:51.856858 27557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 665d270236134191a4ebf76153e0d2c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:51.856959 27557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 665d270236134191a4ebf76153e0d2c3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:51.857857 27557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 665d270236134191a4ebf76153e0d2c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665d270236134191a4ebf76153e0d2c3" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 45879 } }
I20260928 06:39:51.858194 27557 leader_election.cc:302] T 00000000000000000000000000000000 P 665d270236134191a4ebf76153e0d2c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 665d270236134191a4ebf76153e0d2c3; no voters:
I20260928 06:39:51.858505 27557 leader_election.cc:288] T 00000000000000000000000000000000 P 665d270236134191a4ebf76153e0d2c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:51.858598 27567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 665d270236134191a4ebf76153e0d2c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:51.858784 27567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 665d270236134191a4ebf76153e0d2c3 [term 1 LEADER]: Becoming Leader. State: Replica: 665d270236134191a4ebf76153e0d2c3, State: Running, Role: LEADER
I20260928 06:39:51.859175 27567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 665d270236134191a4ebf76153e0d2c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665d270236134191a4ebf76153e0d2c3" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 45879 } }
I20260928 06:39:51.859709 27557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 665d270236134191a4ebf76153e0d2c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:51.860775 27568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 665d270236134191a4ebf76153e0d2c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "665d270236134191a4ebf76153e0d2c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665d270236134191a4ebf76153e0d2c3" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 45879 } } }
I20260928 06:39:51.860926 27568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 665d270236134191a4ebf76153e0d2c3 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:51.861368 27576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:51.861320 27569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 665d270236134191a4ebf76153e0d2c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 665d270236134191a4ebf76153e0d2c3. Latest consensus state: current_term: 1 leader_uuid: "665d270236134191a4ebf76153e0d2c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665d270236134191a4ebf76153e0d2c3" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 45879 } } }
I20260928 06:39:51.861480 27569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 665d270236134191a4ebf76153e0d2c3 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:51.864249 27576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:51.869946 27576 catalog_manager.cc:1347] Generated new cluster ID: 88d60598276243f5bc5e843633d503d0
I20260928 06:39:51.870016 27576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:51.884251 27576 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:51.884594 27576 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:51.894990 27576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 665d270236134191a4ebf76153e0d2c3: Generated new TSK 0
I20260928 06:39:51.895735 27576 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:51.902812 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 27489
2026-09-28T06:39:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1407 ms)
[----------] 18 tests from AuthTokenIssuingTest (25815 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:51 dist-test-slave-j0qf krb5kdc[27588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:51 dist-test-slave-j0qf krb5kdc[27588](info): set up 2 sockets
Sep 28 06:39:51 dist-test-slave-j0qf krb5kdc[27588](info): commencing operation
krb5kdc: starting...
W20260928 06:39:52.941002 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:39:53 dist-test-slave-j0qf krb5kdc[27588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:39:53.070700 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:32827
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:34721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:32827,127.5.22.61:41297,127.5.22.60:46181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:53.180473 27601 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:39:53.180845 27601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:53.180920 27601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:53.180998 27601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:53.185071 27601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:53.185171 27601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:53.185212 27601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:53.185272 27601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:53.185317 27601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:53.190661 27601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:34721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:32827,127.5.22.61:41297,127.5.22.60:46181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:32827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:53.191747 27601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:53.192792 27601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:53.201678 27606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:53.201689 27607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:53.201689 27609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:53.202091 27601 server_base.cc:1034] running on GCE node
I20260928 06:39:53.202785 27601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:53.203941 27601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:53.205212 27601 hybrid_clock.cc:648] HybridClock initialized: now 1790577593205124 us; error 32 us; skew 500 ppm
Sep 28 06:39:53 dist-test-slave-j0qf krb5kdc[27588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577593, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:53.207999 27601 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:39:53.208887 27601 webserver.cc:460] Webserver started at http://127.5.22.62:37047/ using document root <none> and password file <none>
I20260928 06:39:53.209520 27601 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:53.209616 27601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:53.209852 27601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:53.211953 27601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "20cefaf61d734a7b9697c178fc43b1c5"
format_stamp: "Formatted at 2026-09-28 06:39:53 on dist-test-slave-j0qf"
server_key: "521d1540868e6ef1d3979536405bb68f"
server_key_iv: "aa57ebe6228e9fd8fe8acdd1f3453848"
server_key_version: "encryptionkey@0"
I20260928 06:39:53.212544 27601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "20cefaf61d734a7b9697c178fc43b1c5"
format_stamp: "Formatted at 2026-09-28 06:39:53 on dist-test-slave-j0qf"
server_key: "521d1540868e6ef1d3979536405bb68f"
server_key_iv: "aa57ebe6228e9fd8fe8acdd1f3453848"
server_key_version: "encryptionkey@0"
I20260928 06:39:53.216606 27601 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:39:53.219152 27616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:53.220515 27601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:39:53.220670 27601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "20cefaf61d734a7b9697c178fc43b1c5"
format_stamp: "Formatted at 2026-09-28 06:39:53 on dist-test-slave-j0qf"
server_key: "521d1540868e6ef1d3979536405bb68f"
server_key_iv: "aa57ebe6228e9fd8fe8acdd1f3453848"
server_key_version: "encryptionkey@0"
I20260928 06:39:53.220798 27601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:53.271348 27601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:53.274746 27601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:53.274972 27601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:53.282707 27601 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:32827
I20260928 06:39:53.283089 27668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:32827 every 8 connection(s)
I20260928 06:39:53.283774 27601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:53.286799 27669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:53.287633 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 27601
I20260928 06:39:53.287768 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:53.288057 5208 external_mini_cluster.cc:1428] Setting key 78373f6aaca444dbf9bdbf1c6a719ca5
I20260928 06:39:53.291160 27669 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.62" port: 32827 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.22.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.61@KRBTEST.COM" created.
Sep 28 06:39:53 dist-test-slave-j0qf krb5kdc[27588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577593, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:39:53.309056 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.293861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34338 (local address 127.5.22.62:32827)
0928 06:39:53.294346 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:53.294366 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:53.294623 (+ 257us) server_negotiation.cc:407] Connection header received
0928 06:39:53.295713 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:53.295717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:53.295900 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:53.296096 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:53.297933 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.298990 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.300132 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.300744 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.300947 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:53.303846 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:53.303852 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:53.303863 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:53.303883 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:53.305695 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.306209 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.306215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.306217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.306413 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.306737 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.306741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.306743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.307136 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:53.307285 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:53.307652 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 06:39:53.307934 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":143,"threads_started":1}
I20260928 06:39:53.309056 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.293862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:32827 (local address 127.0.0.1:34338)
0928 06:39:53.294472 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.294510 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:53.295389 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:53.296093 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:53.296098 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:53.297480 (+ 1382us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:53.297754 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:53.297770 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.299122 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.299125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:53.299982 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:53.300022 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.300935 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.300939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:53.301059 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:53.301774 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:53.301802 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:53.303679 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:53.305843 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:53.305852 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:53.305863 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:53.306094 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:53.306514 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:53.306519 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:53.306521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:53.306638 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:53.307257 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:53.307263 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:53.307414 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 06:39:53.307591 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 06:39:53.308012 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":107,"thread_start_us":171,"threads_started":1}
I20260928 06:39:53.312007 27669 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.61" port: 41297 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:39:53.312765 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.312501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:41297 (local address 127.0.0.1:49122)
0928 06:39:53.312627 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.312700 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.61:41297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 06:39:53.313001 27619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.22.61:41297: connect: Connection refused (error 111)
I20260928 06:39:53.313942 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.313726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:41297 (local address 127.0.0.1:49138)
0928 06:39:53.313829 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.313884 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.61:41297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 06:39:53.314126 27669 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.61:41297: Network error: Client connection negotiation failed: client connection to 127.5.22.61:41297: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.61.keytab.
Entry for principal kudu/127.5.22.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.61.keytab.
I20260928 06:39:53.317170 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.61:41297
--webserver_interface=127.5.22.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.61.keytab
--principal=kudu/127.5.22.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:34721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:32827,127.5.22.61:41297,127.5.22.60:46181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:39:53.363402 27669 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.61" port: 41297 } attempt: 1
I20260928 06:39:53.364338 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.363996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:41297 (local address 127.0.0.1:49148)
0928 06:39:53.364170 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.364247 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.61:41297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 06:39:53.365113 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.364884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:41297 (local address 127.0.0.1:49160)
0928 06:39:53.364997 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.365048 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.61:41297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 06:39:53.365271 27669 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.61:41297: Network error: Client connection negotiation failed: client connection to 127.5.22.61:41297: connect: Connection refused (error 111)
W20260928 06:39:53.433297 27676 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:39:53.433583 27676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20260928 06:39:53.433472 27669 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.61" port: 41297 } attempt: 2
W20260928 06:39:53.433607 27676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:53.433717 27676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 06:39:53.434394 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.434024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:41297 (local address 127.0.0.1:49168)
0928 06:39:53.434198 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.434274 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.61:41297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 06:39:53.435122 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.434896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:41297 (local address 127.0.0.1:49170)
0928 06:39:53.435011 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.435063 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.61:41297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 06:39:53.435295 27669 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.61:41297: Network error: Client connection negotiation failed: client connection to 127.5.22.61:41297: connect: Connection refused (error 111)
W20260928 06:39:53.438256 27676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:53.438369 27676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:53.438403 27676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:53.438457 27676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:53.438481 27676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:53.443940 27676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:34721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:32827,127.5.22.61:41297,127.5.22.60:46181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.61.keytab
--principal=kudu/127.5.22.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.61:41297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.22.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:53.445036 27676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:53.446105 27676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:53.454756 27682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:53.454758 27681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:53.454816 27676 server_base.cc:1034] running on GCE node
W20260928 06:39:53.454698 27684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:53.455498 27676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:53.456597 27676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:53.457782 27676 hybrid_clock.cc:648] HybridClock initialized: now 1790577593457745 us; error 50 us; skew 500 ppm
Sep 28 06:39:53 dist-test-slave-j0qf krb5kdc[27588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577593, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:53.460529 27676 init.cc:377] Logged in from keytab as kudu/127.5.22.61@KRBTEST.COM (short username kudu)
I20260928 06:39:53.461328 27676 webserver.cc:460] Webserver started at http://127.5.22.61:42533/ using document root <none> and password file <none>
I20260928 06:39:53.461825 27676 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:53.461882 27676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:53.462041 27676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:53.464078 27676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-1/data/instance:
uuid: "0dbf6facf65d4b31b98a149d73f2f884"
format_stamp: "Formatted at 2026-09-28 06:39:53 on dist-test-slave-j0qf"
server_key: "9d02260714114002d51b59bd79e5a3d5"
server_key_iv: "ededaf414302472b201bca21a364a30b"
server_key_version: "encryptionkey@0"
I20260928 06:39:53.464583 27676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-1/wal/instance:
uuid: "0dbf6facf65d4b31b98a149d73f2f884"
format_stamp: "Formatted at 2026-09-28 06:39:53 on dist-test-slave-j0qf"
server_key: "9d02260714114002d51b59bd79e5a3d5"
server_key_iv: "ededaf414302472b201bca21a364a30b"
server_key_version: "encryptionkey@0"
I20260928 06:39:53.468194 27676 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260928 06:39:53.470537 27691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:53.471822 27676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:39:53.471966 27676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-1/wal
uuid: "0dbf6facf65d4b31b98a149d73f2f884"
format_stamp: "Formatted at 2026-09-28 06:39:53 on dist-test-slave-j0qf"
server_key: "9d02260714114002d51b59bd79e5a3d5"
server_key_iv: "ededaf414302472b201bca21a364a30b"
server_key_version: "encryptionkey@0"
I20260928 06:39:53.472078 27676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:53.489238 27676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:53.492461 27676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:53.492691 27676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:53.501104 27676 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.61:41297
I20260928 06:39:53.501462 27743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.61:41297 every 8 connection(s)
I20260928 06:39:53.502242 27676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-1/data/info.pb
I20260928 06:39:53.503599 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 27676
I20260928 06:39:53.503708 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-1/wal/instance
I20260928 06:39:53.504019 5208 external_mini_cluster.cc:1428] Setting key b7280c2d3e3b6a28ff31739753cf89ff
I20260928 06:39:53.505349 27744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:53.509418 27744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.62" port: 32827 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.22.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.60@KRBTEST.COM" created.
Sep 28 06:39:53 dist-test-slave-j0qf krb5kdc[27588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577593, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.61@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:39:53.524672 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.511390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34344 (local address 127.5.22.62:32827)
0928 06:39:53.511576 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:53.511582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:53.512429 (+ 847us) server_negotiation.cc:407] Connection header received
0928 06:39:53.513367 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:53.513370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:53.513416 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:53.513495 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:53.515033 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.515969 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.517658 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.517903 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.518029 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:53.520896 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:53.520900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:53.520902 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:53.520926 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:53.522485 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.523004 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.523008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.523010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.523095 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.523444 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.523449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.523450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.523635 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:53.523741 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:53.524348 (+ 607us) server_negotiation.cc:299] Negotiation successful
0928 06:39:53.524471 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 06:39:53.525487 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.511757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:32827 (local address 127.0.0.1:34344)
0928 06:39:53.512271 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.512314 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:53.513176 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:53.513736 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:53.513744 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:53.514106 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:53.514882 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:53.514897 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.516123 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.516128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:53.517533 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:53.517543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.518074 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.518078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:53.518248 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:53.518939 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:53.518966 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:53.520711 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:53.522612 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:53.522618 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:53.522633 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:53.522897 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:53.523191 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:53.523195 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:53.523197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:53.523344 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:53.523754 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:53.523770 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:53.524075 (+ 305us) client_negotiation.cc:769] Sending connection context
0928 06:39:53.524265 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 06:39:53.524623 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":123,"threads_started":1}
I20260928 06:39:53.526454 27669 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.61" port: 41297 } attempt: 3
I20260928 06:39:53.527204 27744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.61" port: 41297 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.60.keytab.
Entry for principal kudu/127.5.22.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.60.keytab.
I20260928 06:39:53.531971 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.60:46181
--webserver_interface=127.5.22.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.60.keytab
--principal=kudu/127.5.22.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:34721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:32827,127.5.22.61:41297,127.5.22.60:46181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:39:53 dist-test-slave-j0qf krb5kdc[27588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577593, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.61@KRBTEST.COM for kudu/127.5.22.61@KRBTEST.COM
Sep 28 06:39:53 dist-test-slave-j0qf krb5kdc[27588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577593, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for kudu/127.5.22.61@KRBTEST.COM
I20260928 06:39:53.538993 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.527767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:41297 (local address 127.0.0.1:49196)
0928 06:39:53.527896 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.527910 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:53.527970 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:53.528424 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:53.528427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:53.528594 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:53.528677 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:53.528683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.529686 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.529692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:53.530480 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:53.530490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.531195 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.531201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:53.531306 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:53.531936 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:53.531951 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:53.533981 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:53.536943 (+ 2962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:53.536949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:53.536953 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:53.537238 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:53.537820 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:53.537825 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:53.537829 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:53.537945 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:53.538459 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:53.538461 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:53.538546 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 06:39:53.538667 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 06:39:53.538803 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 06:39:53.539065 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.526904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:41297 (local address 127.0.0.1:49182)
0928 06:39:53.527405 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.527421 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:53.527508 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:53.527969 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:53.527971 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:53.528187 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:53.528307 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:53.528312 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.529650 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.529654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:53.530793 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:53.530806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.531484 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.531487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:53.531602 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:53.532081 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:53.532115 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:53.533981 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:53.536970 (+ 2989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:53.536973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:53.536975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:53.537180 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:53.537538 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:53.537543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:53.537547 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:53.537668 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:53.538366 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:53.538372 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:53.538505 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 06:39:53.538644 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 06:39:53.538827 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427}
I20260928 06:39:53.539036 27750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.527860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49196 (local address 127.5.22.61:41297)
0928 06:39:53.528179 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:53.528186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:53.528198 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:39:53.528251 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:53.528253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:53.528302 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:53.528410 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:53.528832 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.529581 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.530590 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.531073 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.531174 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:53.534139 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:53.534142 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:53.534157 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:53.534196 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:53.536822 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.537365 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.537370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.537374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.537439 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.538063 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.538066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.538068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.538380 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:53.538465 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:53.538748 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 06:39:53.538874 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":184,"server-negotiator.queue_time_us":260,"spinlock_wait_cycles":8576,"thread_start_us":115,"threads_started":1}
I20260928 06:39:53.539696 27749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.527005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49182 (local address 127.5.22.61:41297)
0928 06:39:53.527381 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:53.527394 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:53.527474 (+ 80us) server_negotiation.cc:407] Connection header received
0928 06:39:53.527602 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:53.527608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:53.527818 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:53.528428 (+ 610us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:53.528530 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.529527 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.530961 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.531363 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.531471 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:53.534130 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:53.534142 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:53.534157 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:53.534195 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:53.536845 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.537326 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.537332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.537334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.537431 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.537832 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.537835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.537836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.538207 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:53.538377 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:53.539330 (+ 953us) server_negotiation.cc:299] Negotiation successful
0928 06:39:53.539575 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"spinlock_wait_cycles":5760,"thread_start_us":178,"threads_started":1}
I20260928 06:39:53.541353 27669 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:39:53.541456 27744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:39:53.542225 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.541914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:46181 (local address 127.0.0.1:56366)
0928 06:39:53.542070 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.542139 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 06:39:53.542829 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.541914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:46181 (local address 127.0.0.1:56382)
0928 06:39:53.542643 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.542732 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":637}
I20260928 06:39:53.543089 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.542806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:46181 (local address 127.0.0.1:56386)
0928 06:39:53.542961 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.543015 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260928 06:39:53.543251 27669 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.60:46181: Network error: Client connection negotiation failed: client connection to 127.5.22.60:46181: connect: Connection refused (error 111)
W20260928 06:39:53.543346 27695 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.22.60:46181: connect: Connection refused (error 111)
I20260928 06:39:53.544319 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.544054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:46181 (local address 127.0.0.1:56390)
0928 06:39:53.544182 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.544248 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 06:39:53.544473 27744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.60:46181: Network error: Client connection negotiation failed: client connection to 127.5.22.60:46181: connect: Connection refused (error 111)
I20260928 06:39:53.574400 27669 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } attempt: 1
I20260928 06:39:53.575314 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.575011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:46181 (local address 127.0.0.1:56398)
0928 06:39:53.575174 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.575238 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 06:39:53.576121 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.575886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:46181 (local address 127.0.0.1:56404)
0928 06:39:53.575998 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.576054 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 06:39:53.576277 27669 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.60:46181: Network error: Client connection negotiation failed: client connection to 127.5.22.60:46181: connect: Connection refused (error 111)
I20260928 06:39:53.593761 27744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } attempt: 1
I20260928 06:39:53.594666 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.594339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:46181 (local address 127.0.0.1:56406)
0928 06:39:53.594490 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.594562 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 06:39:53.595409 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.595194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:46181 (local address 127.0.0.1:56410)
0928 06:39:53.595298 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.595351 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 06:39:53.595567 27744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.60:46181: Network error: Client connection negotiation failed: client connection to 127.5.22.60:46181: connect: Connection refused (error 111)
I20260928 06:39:53.651436 27669 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } attempt: 2
I20260928 06:39:53.652344 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.651993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:46181 (local address 127.0.0.1:56414)
0928 06:39:53.652182 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.652252 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 06:39:53.653147 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.652874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:46181 (local address 127.0.0.1:56428)
0928 06:39:53.653011 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.653071 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 06:39:53.653330 27669 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.60:46181: Network error: Client connection negotiation failed: client connection to 127.5.22.60:46181: connect: Connection refused (error 111)
W20260928 06:39:53.656047 27751 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:39:53.656303 27751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:53.656356 27751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:53.656445 27751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:53.660493 27751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:53.660542 27751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:53.660561 27751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:53.660604 27751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:53.660640 27751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:53.663754 27744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } attempt: 2
I20260928 06:39:53.664536 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.664217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:46181 (local address 127.0.0.1:56440)
0928 06:39:53.664375 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.664449 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 06:39:53.665298 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.665084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:46181 (local address 127.0.0.1:56454)
0928 06:39:53.665191 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.665240 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 06:39:53.665551 27744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.60:46181: Network error: Client connection negotiation failed: client connection to 127.5.22.60:46181: connect: Connection refused (error 111)
I20260928 06:39:53.666581 27751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:34721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:32827,127.5.22.61:41297,127.5.22.60:46181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.60.keytab
--principal=kudu/127.5.22.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.60:46181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.22.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:53.667678 27751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:53.668773 27751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:53.677965 27760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:53.677950 27757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:53.677970 27758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:53.678452 27751 server_base.cc:1034] running on GCE node
I20260928 06:39:53.679126 27751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:53.680265 27751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:53.681445 27751 hybrid_clock.cc:648] HybridClock initialized: now 1790577593681414 us; error 46 us; skew 500 ppm
Sep 28 06:39:53 dist-test-slave-j0qf krb5kdc[27588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577593, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:53.684412 27751 init.cc:377] Logged in from keytab as kudu/127.5.22.60@KRBTEST.COM (short username kudu)
I20260928 06:39:53.685324 27751 webserver.cc:460] Webserver started at http://127.5.22.60:43005/ using document root <none> and password file <none>
I20260928 06:39:53.685842 27751 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:53.685901 27751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:53.686070 27751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:53.688201 27751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-2/data/instance:
uuid: "4fc8684e3f3e4be8b7352ef7f8d7b6ef"
format_stamp: "Formatted at 2026-09-28 06:39:53 on dist-test-slave-j0qf"
server_key: "f03f347783b1b8ecb3a096d11a4f7bd4"
server_key_iv: "96d068c43ed454383d019a6f49302aae"
server_key_version: "encryptionkey@0"
I20260928 06:39:53.688879 27751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-2/wal/instance:
uuid: "4fc8684e3f3e4be8b7352ef7f8d7b6ef"
format_stamp: "Formatted at 2026-09-28 06:39:53 on dist-test-slave-j0qf"
server_key: "f03f347783b1b8ecb3a096d11a4f7bd4"
server_key_iv: "96d068c43ed454383d019a6f49302aae"
server_key_version: "encryptionkey@0"
I20260928 06:39:53.692767 27751 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 06:39:53.695185 27767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:53.696491 27751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260928 06:39:53.696626 27751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-2/wal
uuid: "4fc8684e3f3e4be8b7352ef7f8d7b6ef"
format_stamp: "Formatted at 2026-09-28 06:39:53 on dist-test-slave-j0qf"
server_key: "f03f347783b1b8ecb3a096d11a4f7bd4"
server_key_iv: "96d068c43ed454383d019a6f49302aae"
server_key_version: "encryptionkey@0"
I20260928 06:39:53.696740 27751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:53.722112 27751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:53.725958 27751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:53.726214 27751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:53.735708 27751 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.60:46181
I20260928 06:39:53.736042 27819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.60:46181 every 8 connection(s)
I20260928 06:39:53.736907 27751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-2/data/info.pb
I20260928 06:39:53.739828 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 27751
I20260928 06:39:53.739989 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577308481727-5208-0/minicluster-data/master-2/wal/instance
I20260928 06:39:53.740083 27820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:53.740347 5208 external_mini_cluster.cc:1428] Setting key da151e5da99b92c6998abcfb306551fe
I20260928 06:39:53.745977 27820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.62" port: 32827 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:39:53 dist-test-slave-j0qf krb5kdc[27588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.60@KRBTEST.COM
I20260928 06:39:53.752502 27669 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } attempt: 3
Sep 28 06:39:53 dist-test-slave-j0qf krb5kdc[27588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577593, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.60@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:39:53.756735 27744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } attempt: 3
I20260928 06:39:53.757649 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.741840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56460 (local address 127.5.22.60:46181)
0928 06:39:53.742259 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:53.742284 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:53.742365 (+ 81us) server_negotiation.cc:407] Connection header received
0928 06:39:53.743322 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:53.743356 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:53.743817 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:53.744249 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:53.744990 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.746713 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.747732 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.748363 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.748531 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:53.751406 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:53.751411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:53.751420 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:53.751447 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:53.753765 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.754255 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.754261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.754267 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.754465 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.754865 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.754871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.754873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.755423 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:53.755668 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:53.755990 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 06:39:53.756251 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":98,"threads_started":1}
I20260928 06:39:53.759299 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.748169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34358 (local address 127.5.22.62:32827)
0928 06:39:53.748327 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:53.748332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:53.748634 (+ 302us) server_negotiation.cc:407] Connection header received
0928 06:39:53.748849 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:53.748853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:53.748910 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:53.749002 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:53.749961 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.750711 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.751697 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.751990 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.752151 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:53.755221 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:53.755224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:53.755227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:53.755252 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:53.757308 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.757850 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.757853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.757855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.757936 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.758273 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.758276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.758278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.758503 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:53.758614 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:53.758992 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 06:39:53.759112 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 06:39:53 dist-test-slave-j0qf krb5kdc[27588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577593, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for kudu/127.5.22.60@KRBTEST.COM
I20260928 06:39:53.759909 27824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.748038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:32827 (local address 127.0.0.1:34358)
0928 06:39:53.748512 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.748566 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:53.748733 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:53.749053 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:53.749059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:53.749628 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:53.749834 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:53.749840 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.750874 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.750878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:53.751560 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:53.751571 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.752140 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.752143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:53.752229 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:53.752826 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:53.752845 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:53.755070 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:53.757412 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:53.757421 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:53.757427 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:53.757752 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:53.758060 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:53.758063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:53.758064 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:53.758149 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:53.758614 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:53.758621 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:53.758753 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 06:39:53.759498 (+ 745us) client_negotiation.cc:240] Negotiation successful
0928 06:39:53.759696 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":164,"threads_started":1}
I20260928 06:39:53.761293 27820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.61" port: 41297 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:39:53.765054 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.752963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:46181 (local address 127.0.0.1:56466)
0928 06:39:53.753142 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.753156 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:53.754444 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:53.754808 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:53.754811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:53.755001 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:53.755095 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:53.755102 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.756208 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.756214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:53.757033 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:53.757046 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.757792 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.757798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:53.757895 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:53.758669 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:53.758682 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:53.760818 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:53.762909 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:53.762916 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:53.762920 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:53.763183 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:53.763495 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:53.763500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:53.763503 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:53.763603 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:53.764543 (+ 940us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:53.764549 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:53.764647 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 06:39:53.764735 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:39:53.764860 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
Sep 28 06:39:53 dist-test-slave-j0qf krb5kdc[27588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577593, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.61@KRBTEST.COM for kudu/127.5.22.60@KRBTEST.COM
I20260928 06:39:53.765657 27825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.753041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56466 (local address 127.5.22.60:46181)
0928 06:39:53.753317 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:53.753327 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:53.753353 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:39:53.754574 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:53.754577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:53.754626 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:53.754733 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:53.755226 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.755951 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.757196 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.757647 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.758085 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:53.760950 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:53.760954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:53.760956 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:53.760981 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:53.762803 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.763283 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.763287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.763289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.763392 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.763709 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.763714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.763722 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.764189 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:53.765237 (+ 1048us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:53.765312 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 06:39:53.765430 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":118,"threads_started":1}
Sep 28 06:39:53 dist-test-slave-j0qf krb5kdc[27588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
Sep 28 06:39:53 dist-test-slave-j0qf krb5kdc[27588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577593, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.60@KRBTEST.COM for kudu/127.5.22.61@KRBTEST.COM
I20260928 06:39:53.769325 27669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5: Bootstrap starting.
I20260928 06:39:53.770043 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.757188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:46181 (local address 127.0.0.1:56470)
0928 06:39:53.757450 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.757466 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:53.757555 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:53.758759 (+ 1204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:53.758762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:53.758969 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:53.759080 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:53.759088 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.761527 (+ 2439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.761531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:53.762427 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:53.762438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.763603 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.763606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:53.763701 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:53.764126 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:53.764143 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:53.765944 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:53.768308 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:53.768312 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:53.768314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:53.768568 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:53.768895 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:53.768900 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:53.768902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:53.769010 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:53.769501 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:53.769505 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:53.769603 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 06:39:53.769726 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 06:39:53.769869 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260928 06:39:53.770054 27826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.757418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56470 (local address 127.5.22.60:46181)
0928 06:39:53.758512 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:53.758521 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:53.758533 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:39:53.758565 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:53.758568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:53.758607 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:53.759196 (+ 589us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:53.759298 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.761154 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.763094 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.763480 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.763585 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:53.766377 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:53.766382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:53.766387 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:53.766411 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:53.768150 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.768688 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.768691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.768693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.768780 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.769114 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.769117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.769119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.769330 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:53.769435 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:53.769798 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 06:39:53.769908 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1030,"thread_start_us":107,"threads_started":1}
I20260928 06:39:53.772404 27669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:53.773329 27744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dbf6facf65d4b31b98a149d73f2f884: Bootstrap starting.
I20260928 06:39:53.773453 27669 log.cc:824] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:53.773629 27824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.762689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:41297 (local address 127.0.0.1:49210)
0928 06:39:53.764053 (+ 1364us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.764066 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:53.764178 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:53.764584 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:53.764587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:53.764765 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:53.764886 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:53.764894 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.766080 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.766083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:53.767006 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:53.767016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.767680 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.767684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:53.767862 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:53.768297 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:53.768312 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:53.769751 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:53.771784 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:53.771787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:53.771789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:53.772038 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:53.772381 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:53.772385 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:53.772390 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:53.772538 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:53.773070 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:53.773074 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:53.773166 (+ 92us) client_negotiation.cc:769] Sending connection context
0928 06:39:53.773280 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 06:39:53.773464 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1271}
I20260928 06:39:53.773669 27749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.762847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.5.22.61:41297)
0928 06:39:53.762998 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:53.763005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:53.764117 (+ 1112us) server_negotiation.cc:407] Connection header received
0928 06:39:53.764314 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:53.764320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:53.764376 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:53.764465 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:53.765018 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.765749 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.767131 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.767545 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.767668 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:53.769916 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:53.769919 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:53.769922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:53.769946 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:53.771663 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.772155 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.772167 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.772171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.772263 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.772634 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.772639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.772642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.772952 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:53.773053 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:53.773358 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 06:39:53.773501 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:39:53.775058 27820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:39:53.776419 27744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dbf6facf65d4b31b98a149d73f2f884: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:53.776671 27669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5: No bootstrap required, opened a new log
I20260928 06:39:53.777498 27744 log.cc:824] T 00000000000000000000000000000000 P 0dbf6facf65d4b31b98a149d73f2f884: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:53.778532 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.760538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34374 (local address 127.5.22.62:32827)
0928 06:39:53.760631 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:53.760635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:53.760651 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:39:53.760694 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:53.760698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:53.760740 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:53.760818 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:53.761738 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.762511 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.763484 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.763783 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.763891 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:53.774349 (+ 10458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:53.774356 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:53.774360 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:53.774385 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:53.776139 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.776760 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.776763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.776765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.776875 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.777244 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.777247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.777249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.777448 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:53.777600 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:53.778249 (+ 649us) server_negotiation.cc:299] Negotiation successful
0928 06:39:53.778388 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:39:53.780082 27744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dbf6facf65d4b31b98a149d73f2f884: No bootstrap required, opened a new log
Sep 28 06:39:53 dist-test-slave-j0qf krb5kdc[27588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577593, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.60@KRBTEST.COM for kudu/127.5.22.60@KRBTEST.COM
I20260928 06:39:53.781665 27669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cefaf61d734a7b9697c178fc43b1c5" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 32827 } } peers { permanent_uuid: "0dbf6facf65d4b31b98a149d73f2f884" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 41297 } } peers { permanent_uuid: "4fc8684e3f3e4be8b7352ef7f8d7b6ef" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } }
I20260928 06:39:53.782256 27669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20cefaf61d734a7b9697c178fc43b1c5, State: Initialized, Role: FOLLOWER
I20260928 06:39:53.782693 27669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cefaf61d734a7b9697c178fc43b1c5" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 32827 } } peers { permanent_uuid: "0dbf6facf65d4b31b98a149d73f2f884" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 41297 } } peers { permanent_uuid: "4fc8684e3f3e4be8b7352ef7f8d7b6ef" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } }
I20260928 06:39:53.784266 27669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:53.784268 27744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dbf6facf65d4b31b98a149d73f2f884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cefaf61d734a7b9697c178fc43b1c5" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 32827 } } peers { permanent_uuid: "0dbf6facf65d4b31b98a149d73f2f884" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 41297 } } peers { permanent_uuid: "4fc8684e3f3e4be8b7352ef7f8d7b6ef" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } }
I20260928 06:39:53.784479 27744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dbf6facf65d4b31b98a149d73f2f884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dbf6facf65d4b31b98a149d73f2f884, State: Initialized, Role: FOLLOWER
I20260928 06:39:53.784806 27744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dbf6facf65d4b31b98a149d73f2f884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cefaf61d734a7b9697c178fc43b1c5" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 32827 } } peers { permanent_uuid: "0dbf6facf65d4b31b98a149d73f2f884" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 41297 } } peers { permanent_uuid: "4fc8684e3f3e4be8b7352ef7f8d7b6ef" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } }
I20260928 06:39:53.785027 27826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.775735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56474 (local address 127.5.22.60:46181)
0928 06:39:53.775851 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:53.775856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:53.776054 (+ 198us) server_negotiation.cc:407] Connection header received
0928 06:39:53.776217 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:53.776220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:53.776268 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:53.776319 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:53.776932 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.777724 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.778772 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.779136 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.779225 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:53.781567 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:53.781570 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:53.781572 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:53.781592 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:53.783255 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.783674 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.783677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.783679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.783753 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.784044 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.784050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.784054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.784289 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:53.784364 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:53.784703 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 06:39:53.784827 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:39:53.785507 27824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.775532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:46181 (local address 127.0.0.1:56474)
0928 06:39:53.775991 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:53.776007 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:53.776109 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:53.776433 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:53.776436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:53.776593 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:53.776803 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:53.776811 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.777842 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.777848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:53.778653 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:53.778664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.779352 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.779355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:53.779424 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:53.779996 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:53.780012 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:53.781447 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:53.783353 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:53.783356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:53.783358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:53.783581 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:53.783851 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:53.783854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:53.783859 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:53.783951 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:53.784429 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:53.784432 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:53.784521 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 06:39:53.785211 (+ 690us) client_negotiation.cc:240] Negotiation successful
0928 06:39:53.785349 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336}
Sep 28 06:39:53 dist-test-slave-j0qf krb5kdc[27588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.61@KRBTEST.COM
I20260928 06:39:53.786203 27744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dbf6facf65d4b31b98a149d73f2f884 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:53.788677 27820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc8684e3f3e4be8b7352ef7f8d7b6ef: Bootstrap starting.
I20260928 06:39:53.790462 27829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cefaf61d734a7b9697c178fc43b1c5" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 32827 } } peers { permanent_uuid: "0dbf6facf65d4b31b98a149d73f2f884" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 41297 } } peers { permanent_uuid: "4fc8684e3f3e4be8b7352ef7f8d7b6ef" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } } }
I20260928 06:39:53.790628 27829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:39:53.791409 27820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fc8684e3f3e4be8b7352ef7f8d7b6ef: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:53.792279 27820 log.cc:824] T 00000000000000000000000000000000 P 4fc8684e3f3e4be8b7352ef7f8d7b6ef: Log is configured to *not* fsync() on all Append() calls
W20260928 06:39:53.792520 27852 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0dbf6facf65d4b31b98a149d73f2f884: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:39:53.792637 27852 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 06:39:53.793032 27842 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:39:53.793104 27842 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:39:53.789532 27830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dbf6facf65d4b31b98a149d73f2f884 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cefaf61d734a7b9697c178fc43b1c5" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 32827 } } peers { permanent_uuid: "0dbf6facf65d4b31b98a149d73f2f884" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 41297 } } peers { permanent_uuid: "4fc8684e3f3e4be8b7352ef7f8d7b6ef" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } } }
I20260928 06:39:53.793427 27830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dbf6facf65d4b31b98a149d73f2f884 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:39:53.793962 27601 master_runner.cc:186] Error getting master registration for 127.5.22.60:46181: OK, instance_id { permanent_uuid: "4fc8684e3f3e4be8b7352ef7f8d7b6ef" instance_seqno: 1790577593733038 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:39:53.797335 27601 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:39:53.799556 27820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc8684e3f3e4be8b7352ef7f8d7b6ef: No bootstrap required, opened a new log
I20260928 06:39:53.799121 27676 master_runner.cc:186] Error getting master registration for 127.5.22.60:46181: OK, instance_id { permanent_uuid: "4fc8684e3f3e4be8b7352ef7f8d7b6ef" instance_seqno: 1790577593733038 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:39:53.802681 27676 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:39:53.804281 27749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.779896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49222 (local address 127.5.22.61:41297)
0928 06:39:53.780002 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:53.780006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:53.780017 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:39:53.780053 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:53.780057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:53.780098 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:53.780174 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:53.781047 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.781782 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.783062 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.783425 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.783500 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:53.786453 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:53.786457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:53.786459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:53.786485 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:53.788133 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.798333 (+ 10200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.798337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.798340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.798443 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.803065 (+ 4622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.803069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.803071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.803345 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:53.803561 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:53.803962 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 06:39:53.804094 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:39:53.804441 27820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fc8684e3f3e4be8b7352ef7f8d7b6ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cefaf61d734a7b9697c178fc43b1c5" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 32827 } } peers { permanent_uuid: "0dbf6facf65d4b31b98a149d73f2f884" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 41297 } } peers { permanent_uuid: "4fc8684e3f3e4be8b7352ef7f8d7b6ef" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } }
I20260928 06:39:53.804724 27820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fc8684e3f3e4be8b7352ef7f8d7b6ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc8684e3f3e4be8b7352ef7f8d7b6ef, State: Initialized, Role: FOLLOWER
I20260928 06:39:53.805085 27820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fc8684e3f3e4be8b7352ef7f8d7b6ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cefaf61d734a7b9697c178fc43b1c5" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 32827 } } peers { permanent_uuid: "0dbf6facf65d4b31b98a149d73f2f884" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 41297 } } peers { permanent_uuid: "4fc8684e3f3e4be8b7352ef7f8d7b6ef" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } }
I20260928 06:39:53.806954 27820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fc8684e3f3e4be8b7352ef7f8d7b6ef [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:53.807786 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:53.809370 27643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "20cefaf61d734a7b9697c178fc43b1c5"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34374
I20260928 06:39:53.809633 27643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 06:39:53.809777 27643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:53.811331 27643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cefaf61d734a7b9697c178fc43b1c5" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 32827 } } peers { permanent_uuid: "0dbf6facf65d4b31b98a149d73f2f884" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 41297 } } peers { permanent_uuid: "4fc8684e3f3e4be8b7352ef7f8d7b6ef" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } }
I20260928 06:39:53.812311 27643 leader_election.cc:288] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5 [CANDIDATE]: Term 1 election: Requested vote from peers 0dbf6facf65d4b31b98a149d73f2f884 (127.5.22.61:41297), 4fc8684e3f3e4be8b7352ef7f8d7b6ef (127.5.22.60:46181)
I20260928 06:39:53.812799 27794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "20cefaf61d734a7b9697c178fc43b1c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4fc8684e3f3e4be8b7352ef7f8d7b6ef"
I20260928 06:39:53.813021 27794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fc8684e3f3e4be8b7352ef7f8d7b6ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:53.813225 27717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "20cefaf61d734a7b9697c178fc43b1c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0dbf6facf65d4b31b98a149d73f2f884"
I20260928 06:39:53.813457 27717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dbf6facf65d4b31b98a149d73f2f884 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:53.814558 27794 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4fc8684e3f3e4be8b7352ef7f8d7b6ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20cefaf61d734a7b9697c178fc43b1c5 in term 1.
I20260928 06:39:53.814926 27717 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0dbf6facf65d4b31b98a149d73f2f884 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20cefaf61d734a7b9697c178fc43b1c5 in term 1.
I20260928 06:39:53.814980 27620 leader_election.cc:302] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20cefaf61d734a7b9697c178fc43b1c5, 4fc8684e3f3e4be8b7352ef7f8d7b6ef; no voters:
I20260928 06:39:53.815297 27829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:53.815521 27829 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5 [term 1 LEADER]: Becoming Leader. State: Replica: 20cefaf61d734a7b9697c178fc43b1c5, State: Running, Role: LEADER
I20260928 06:39:53.815855 27829 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cefaf61d734a7b9697c178fc43b1c5" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 32827 } } peers { permanent_uuid: "0dbf6facf65d4b31b98a149d73f2f884" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 41297 } } peers { permanent_uuid: "4fc8684e3f3e4be8b7352ef7f8d7b6ef" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } }
I20260928 06:39:53.817651 27829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20cefaf61d734a7b9697c178fc43b1c5. Latest consensus state: current_term: 1 leader_uuid: "20cefaf61d734a7b9697c178fc43b1c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cefaf61d734a7b9697c178fc43b1c5" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 32827 } } peers { permanent_uuid: "0dbf6facf65d4b31b98a149d73f2f884" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 41297 } } peers { permanent_uuid: "4fc8684e3f3e4be8b7352ef7f8d7b6ef" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } } }
I20260928 06:39:53.817763 27829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5 [sys.catalog]: This master's current role is: LEADER
W20260928 06:39:53.820405 27867 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4fc8684e3f3e4be8b7352ef7f8d7b6ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:39:53.820484 27867 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:39:53.824693 27876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:53.825870 27856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc8684e3f3e4be8b7352ef7f8d7b6ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cefaf61d734a7b9697c178fc43b1c5" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 32827 } } peers { permanent_uuid: "0dbf6facf65d4b31b98a149d73f2f884" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 41297 } } peers { permanent_uuid: "4fc8684e3f3e4be8b7352ef7f8d7b6ef" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } } }
I20260928 06:39:53.826000 27856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc8684e3f3e4be8b7352ef7f8d7b6ef [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:39:53.827723 27876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:53.838305 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.817479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34388 (local address 127.5.22.62:32827)
0928 06:39:53.817592 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:53.817597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:53.820689 (+ 3092us) server_negotiation.cc:407] Connection header received
0928 06:39:53.820889 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:53.820892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:53.820937 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:53.820989 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:53.822927 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.823639 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.827724 (+ 4085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.828026 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.828104 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:53.832181 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:53.832186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:53.832189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:53.832217 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:53.833982 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.834523 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.834527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.834529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.834618 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.834968 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.834972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.834974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.835195 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:53.835311 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:53.838009 (+ 2698us) server_negotiation.cc:299] Negotiation successful
0928 06:39:53.838138 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:39:53.841518 27826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.818144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56480 (local address 127.5.22.60:46181)
0928 06:39:53.818277 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:53.818283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:53.822205 (+ 3922us) server_negotiation.cc:407] Connection header received
0928 06:39:53.822423 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:53.822426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:53.822470 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:53.822518 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:53.825810 (+ 3292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.826544 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.829938 (+ 3394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.830297 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.830390 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:53.835649 (+ 5259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:53.835656 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:53.835674 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:53.835702 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:53.837432 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.839964 (+ 2532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.839967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.839969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.840066 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.840398 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.840402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.840404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.840670 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:53.840801 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:53.841155 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 06:39:53.841304 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:39:53.842419 27794 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4fc8684e3f3e4be8b7352ef7f8d7b6ef [term 1 FOLLOWER]: Refusing update from remote peer 20cefaf61d734a7b9697c178fc43b1c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:53.842522 27717 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0dbf6facf65d4b31b98a149d73f2f884 [term 1 FOLLOWER]: Refusing update from remote peer 20cefaf61d734a7b9697c178fc43b1c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:53.843066 27829 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc8684e3f3e4be8b7352ef7f8d7b6ef" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:53.843485 27829 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dbf6facf65d4b31b98a149d73f2f884" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 41297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:53.847239 27856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc8684e3f3e4be8b7352ef7f8d7b6ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20cefaf61d734a7b9697c178fc43b1c5. Latest consensus state: current_term: 1 leader_uuid: "20cefaf61d734a7b9697c178fc43b1c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cefaf61d734a7b9697c178fc43b1c5" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 32827 } } peers { permanent_uuid: "0dbf6facf65d4b31b98a149d73f2f884" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 41297 } } peers { permanent_uuid: "4fc8684e3f3e4be8b7352ef7f8d7b6ef" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } } }
I20260928 06:39:53.847360 27856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc8684e3f3e4be8b7352ef7f8d7b6ef [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:39:53.854655 27830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dbf6facf65d4b31b98a149d73f2f884 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20cefaf61d734a7b9697c178fc43b1c5. Latest consensus state: current_term: 1 leader_uuid: "20cefaf61d734a7b9697c178fc43b1c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cefaf61d734a7b9697c178fc43b1c5" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 32827 } } peers { permanent_uuid: "0dbf6facf65d4b31b98a149d73f2f884" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 41297 } } peers { permanent_uuid: "4fc8684e3f3e4be8b7352ef7f8d7b6ef" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } } }
I20260928 06:39:53.854801 27830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dbf6facf65d4b31b98a149d73f2f884 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:39:53.855211 27749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.818898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49238 (local address 127.5.22.61:41297)
0928 06:39:53.819008 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:53.819013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:53.826368 (+ 7355us) server_negotiation.cc:407] Connection header received
0928 06:39:53.826523 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:53.826526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:53.826574 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:53.826631 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:53.842427 (+ 15796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.843137 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.844278 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:53.844595 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:53.844672 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:53.847017 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:53.847022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:53.847024 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:53.847052 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:53.849062 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.850398 (+ 1336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.850402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.850404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.850491 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:53.850806 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:53.850810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:53.850812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:53.851032 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:53.851167 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:53.854950 (+ 3783us) server_negotiation.cc:299] Negotiation successful
0928 06:39:53.855065 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:39:53.863306 27829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20cefaf61d734a7b9697c178fc43b1c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cefaf61d734a7b9697c178fc43b1c5" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 32827 } } peers { permanent_uuid: "0dbf6facf65d4b31b98a149d73f2f884" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 41297 } } peers { permanent_uuid: "4fc8684e3f3e4be8b7352ef7f8d7b6ef" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } } }
I20260928 06:39:53.863440 27829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:53.864493 27830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dbf6facf65d4b31b98a149d73f2f884 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20cefaf61d734a7b9697c178fc43b1c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cefaf61d734a7b9697c178fc43b1c5" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 32827 } } peers { permanent_uuid: "0dbf6facf65d4b31b98a149d73f2f884" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 41297 } } peers { permanent_uuid: "4fc8684e3f3e4be8b7352ef7f8d7b6ef" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } } }
I20260928 06:39:53.864614 27830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dbf6facf65d4b31b98a149d73f2f884 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:39:53.865195 27879 mvcc.cc:205] Tried to move back new op lower bound from 7334205824369606656 to 7334205824273252352. Current Snapshot: MvccSnapshot[applied={T|T < 7334205824369606656}]
I20260928 06:39:53.866204 27882 mvcc.cc:205] Tried to move back new op lower bound from 7334205824369606656 to 7334205824273252352. Current Snapshot: MvccSnapshot[applied={T|T < 7334205824369606656}]
I20260928 06:39:53.866535 27876 catalog_manager.cc:1347] Generated new cluster ID: 9c6a25922ff346b68e0d35fc47e457f9
I20260928 06:39:53.866600 27876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:53.869915 27869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20cefaf61d734a7b9697c178fc43b1c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cefaf61d734a7b9697c178fc43b1c5" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 32827 } } peers { permanent_uuid: "0dbf6facf65d4b31b98a149d73f2f884" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 41297 } } peers { permanent_uuid: "4fc8684e3f3e4be8b7352ef7f8d7b6ef" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } } }
I20260928 06:39:53.870074 27869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:53.871479 27856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc8684e3f3e4be8b7352ef7f8d7b6ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20cefaf61d734a7b9697c178fc43b1c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cefaf61d734a7b9697c178fc43b1c5" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 32827 } } peers { permanent_uuid: "0dbf6facf65d4b31b98a149d73f2f884" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 41297 } } peers { permanent_uuid: "4fc8684e3f3e4be8b7352ef7f8d7b6ef" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 46181 } } }
I20260928 06:39:53.871613 27856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc8684e3f3e4be8b7352ef7f8d7b6ef [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:39:53.871937 27880 mvcc.cc:205] Tried to move back new op lower bound from 7334205824369606656 to 7334205824273252352. Current Snapshot: MvccSnapshot[applied={T|T < 7334205824369606656}]
I20260928 06:39:53.885669 27876 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:53.887218 27876 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:53.909786 27876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20cefaf61d734a7b9697c178fc43b1c5: Generated new TSK 0
I20260928 06:39:53.910799 27876 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:39:53.947963 27672 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:53.944398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34404 (local address 127.5.22.62:32827)
0928 06:39:53.944531 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:53.944535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:53.946568 (+ 2033us) server_negotiation.cc:407] Connection header received
0928 06:39:53.947010 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:53.947014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:53.947071 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:53.947127 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:53.947814 (+ 687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34404: BlockingRecv error: recv got EOF from 127.0.0.1:34404 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 06:39:53.949385 27826 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:53.945188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56490 (local address 127.5.22.60:46181)
0928 06:39:53.945312 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:53.945316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:53.946924 (+ 1608us) server_negotiation.cc:407] Connection header received
0928 06:39:53.948564 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:53.948567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:53.948621 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:53.948684 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:53.949252 (+ 568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56490: BlockingRecv error: recv got EOF from 127.0.0.1:56490 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 06:39:53.950531 27749 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:53.945901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49254 (local address 127.5.22.61:41297)
0928 06:39:53.946018 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:53.946023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:53.948471 (+ 2448us) server_negotiation.cc:407] Connection header received
0928 06:39:53.949757 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:53.949761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:53.949813 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:53.949863 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:53.950408 (+ 545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49254: BlockingRecv error: recv got EOF from 127.0.0.1:49254 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 06:39:53.955770 27749 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:53.954147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49262 (local address 127.5.22.61:41297)
0928 06:39:53.954263 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:53.954266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:53.954755 (+ 489us) server_negotiation.cc:407] Connection header received
0928 06:39:53.954979 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:53.954982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:53.955058 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:53.955109 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:53.955624 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49262: BlockingRecv error: recv got EOF from 127.0.0.1:49262 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 06:39:53.960902 27749 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:53.959369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49266 (local address 127.5.22.61:41297)
0928 06:39:53.959473 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:53.959477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:53.960012 (+ 535us) server_negotiation.cc:407] Connection header received
0928 06:39:53.960195 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:53.960199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:53.960248 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:53.960299 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:53.960800 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49266: BlockingRecv error: recv got EOF from 127.0.0.1:49266 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 06:39:53.966254 27749 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:53.964809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49282 (local address 127.5.22.61:41297)
0928 06:39:53.964906 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:53.964910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:53.965392 (+ 482us) server_negotiation.cc:407] Connection header received
0928 06:39:53.965558 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:53.965562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:53.965607 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:53.965655 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:53.966157 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49282: BlockingRecv error: recv got EOF from 127.0.0.1:49282 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 06:39:53.974016 27749 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:53.972375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49288 (local address 127.5.22.61:41297)
0928 06:39:53.972501 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:53.972505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:53.973056 (+ 551us) server_negotiation.cc:407] Connection header received
0928 06:39:53.973252 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:53.973256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:53.973306 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:53.973358 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:53.973905 (+ 547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49288: BlockingRecv error: recv got EOF from 127.0.0.1:49288 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 06:39:53.986826 27749 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:53.984815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.5.22.61:41297)
0928 06:39:53.984945 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:53.984951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:53.985640 (+ 689us) server_negotiation.cc:407] Connection header received
0928 06:39:53.985897 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:53.985901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:53.985958 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:53.986028 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:53.986703 (+ 675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49298: BlockingRecv error: recv got EOF from 127.0.0.1:49298 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 06:39:54.007541 27749 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:54.005532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49308 (local address 127.5.22.61:41297)
0928 06:39:54.005685 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:54.005691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:54.006387 (+ 696us) server_negotiation.cc:407] Connection header received
0928 06:39:54.006649 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:54.006653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:54.006716 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:54.006777 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:54.007416 (+ 639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49308: BlockingRecv error: recv got EOF from 127.0.0.1:49308 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 06:39:54.043783 27749 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:54.042377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49320 (local address 127.5.22.61:41297)
0928 06:39:54.042530 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:54.042533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:54.042759 (+ 226us) server_negotiation.cc:407] Connection header received
0928 06:39:54.042905 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:54.042907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:54.042952 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:54.043034 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:54.043607 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49320: BlockingRecv error: recv got EOF from 127.0.0.1:49320 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 06:39:54.111373 27749 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:54.109937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49326 (local address 127.5.22.61:41297)
0928 06:39:54.110131 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:54.110136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:54.110403 (+ 267us) server_negotiation.cc:407] Connection header received
0928 06:39:54.110603 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:54.110611 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:54.110660 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:54.110756 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:54.111234 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49326: BlockingRecv error: recv got EOF from 127.0.0.1:49326 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 06:39:54.242619 27749 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:54.241509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.5.22.61:41297)
0928 06:39:54.241668 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:54.241672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:54.241715 (+ 43us) server_negotiation.cc:407] Connection header received
0928 06:39:54.241870 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:54.241873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:54.241927 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:54.242018 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:54.242505 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49330: BlockingRecv error: recv got EOF from 127.0.0.1:49330 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 06:39:54.502291 27749 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:54.501016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49332 (local address 127.5.22.61:41297)
0928 06:39:54.501277 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:54.501282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:54.501301 (+ 19us) server_negotiation.cc:407] Connection header received
0928 06:39:54.501456 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:54.501459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:54.501520 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:54.501648 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:54.502187 (+ 539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49332: BlockingRecv error: recv got EOF from 127.0.0.1:49332 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 06:39:54.794129 27852 catalog_manager.cc:1259] Loaded cluster ID: 9c6a25922ff346b68e0d35fc47e457f9
I20260928 06:39:54.794250 27852 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0dbf6facf65d4b31b98a149d73f2f884: loading cluster ID for follower catalog manager: success
I20260928 06:39:54.795985 27852 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0dbf6facf65d4b31b98a149d73f2f884: acquiring CA information for follower catalog manager: success
I20260928 06:39:54.797248 27852 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0dbf6facf65d4b31b98a149d73f2f884: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:39:54.821842 27867 catalog_manager.cc:1259] Loaded cluster ID: 9c6a25922ff346b68e0d35fc47e457f9
I20260928 06:39:54.821941 27867 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4fc8684e3f3e4be8b7352ef7f8d7b6ef: loading cluster ID for follower catalog manager: success
I20260928 06:39:54.823779 27867 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4fc8684e3f3e4be8b7352ef7f8d7b6ef: acquiring CA information for follower catalog manager: success
I20260928 06:39:54.825035 27867 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4fc8684e3f3e4be8b7352ef7f8d7b6ef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:39:55.020866 27952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.016766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49336 (local address 127.5.22.61:41297)
0928 06:39:55.017094 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.017100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.017118 (+ 18us) server_negotiation.cc:407] Connection header received
0928 06:39:55.017209 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.017213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.017347 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.017473 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:55.017761 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.018303 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.019306 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.019543 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.019658 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.019994 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:55.020440 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:55.020587 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:55.020631 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 06:39:55.020689 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":79,"threads_started":1}
I20260928 06:39:55.027848 27958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.024316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56498 (local address 127.5.22.60:46181)
0928 06:39:55.024552 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.024555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.024569 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:39:55.024686 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.024690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.024807 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.024930 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:55.025066 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.025596 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.026457 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.026684 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.026766 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.027060 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:55.027507 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:55.027601 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:55.027675 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 06:39:55.027722 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":86,"threads_started":1}
I20260928 06:39:55.034930 27952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.031120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49340 (local address 127.5.22.61:41297)
0928 06:39:55.031254 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.031258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.031270 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:39:55.031319 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.031322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.031363 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.031452 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:55.032066 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.032757 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.033822 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.034036 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.034150 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.034435 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:55.034515 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:55.034589 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:55.034713 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 06:39:55.034764 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 06:39:55.036090 27965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.030873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34414 (local address 127.5.22.62:32827)
0928 06:39:55.031167 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.031173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.031557 (+ 384us) server_negotiation.cc:407] Connection header received
0928 06:39:55.031846 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.031850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.032022 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.032160 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:55.032512 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.033134 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.034279 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.034628 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.034722 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.035248 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:55.035562 (+ 314us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:55.035684 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:55.035888 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 06:39:55.035944 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":85,"threads_started":1}
I20260928 06:39:55.036463 27958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.031592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56512 (local address 127.5.22.60:46181)
0928 06:39:55.031721 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.031724 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.031754 (+ 30us) server_negotiation.cc:407] Connection header received
0928 06:39:55.031793 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.031796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.031841 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.031915 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:55.032153 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.032765 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.034951 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.035244 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.035363 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.035973 (+ 610us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:55.036058 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:55.036149 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:55.036285 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 06:39:55.036342 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:39:55.039168 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 27601
I20260928 06:39:55.046046 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 27676
I20260928 06:39:55.052376 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 27751
2026-09-28T06:39:55Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3151 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:55 dist-test-slave-j0qf krb5kdc[27969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:55 dist-test-slave-j0qf krb5kdc[27969](info): set up 2 sockets
Sep 28 06:39:55 dist-test-slave-j0qf krb5kdc[27969](info): commencing operation
krb5kdc: starting...
W20260928 06:39:56.113306 5208 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/krb5kdc/test-user.keytab.
Sep 28 06:39:56 dist-test-slave-j0qf krb5kdc[27969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.22.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
Entry for principal kudu/127.5.22.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab.
I20260928 06:39:56.241957 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:35705
--webserver_interface=127.5.22.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:45597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:35705,127.5.22.61:36563,127.5.22.60:45023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:56.354911 27982 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:39:56.355208 27982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:56.355266 27982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:56.355345 27982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:56.359555 27982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:56.359611 27982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:56.359630 27982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:56.359647 27982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:56.359700 27982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:56.365732 27982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:45597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:35705,127.5.22.61:36563,127.5.22.60:45023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.62.keytab
--principal=kudu/127.5.22.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.62:35705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.22.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:56.367036 27982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:56.368175 27982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:56.377301 27987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:56.377341 27990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:56.377476 27982 server_base.cc:1034] running on GCE node
W20260928 06:39:56.377725 27988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:56.378249 27982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:56.379462 27982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:56.380623 27982 hybrid_clock.cc:648] HybridClock initialized: now 1790577596380597 us; error 40 us; skew 500 ppm
Sep 28 06:39:56 dist-test-slave-j0qf krb5kdc[27969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577596, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:56.383569 27982 init.cc:377] Logged in from keytab as kudu/127.5.22.62@KRBTEST.COM (short username kudu)
I20260928 06:39:56.384610 27982 webserver.cc:460] Webserver started at http://127.5.22.62:38139/ using document root <none> and password file <none>
I20260928 06:39:56.385180 27982 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:56.385273 27982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:56.385507 27982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:56.387632 27982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-0/data/instance:
uuid: "f90a1a53145347598f2e1e48964bec8c"
format_stamp: "Formatted at 2026-09-28 06:39:56 on dist-test-slave-j0qf"
server_key: "18472a78616bbba83b93110048724db5"
server_key_iv: "1ece2d9bebf8e72b92598ed08c7877eb"
server_key_version: "encryptionkey@0"
I20260928 06:39:56.388250 27982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-0/wal/instance:
uuid: "f90a1a53145347598f2e1e48964bec8c"
format_stamp: "Formatted at 2026-09-28 06:39:56 on dist-test-slave-j0qf"
server_key: "18472a78616bbba83b93110048724db5"
server_key_iv: "1ece2d9bebf8e72b92598ed08c7877eb"
server_key_version: "encryptionkey@0"
I20260928 06:39:56.391990 27982 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 06:39:56.394589 27997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:56.395840 27982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 06:39:56.395982 27982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-0/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-0/wal
uuid: "f90a1a53145347598f2e1e48964bec8c"
format_stamp: "Formatted at 2026-09-28 06:39:56 on dist-test-slave-j0qf"
server_key: "18472a78616bbba83b93110048724db5"
server_key_iv: "1ece2d9bebf8e72b92598ed08c7877eb"
server_key_version: "encryptionkey@0"
I20260928 06:39:56.396095 27982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:56.417639 27982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:56.421247 27982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:56.421464 27982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:56.429037 27982 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.62:35705
I20260928 06:39:56.429338 28049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.62:35705 every 8 connection(s)
I20260928 06:39:56.430053 27982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:56.432919 28050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:56.436905 28050 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:39:56.438714 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 27982
I20260928 06:39:56.438886 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-0/wal/instance
I20260928 06:39:56.439231 5208 external_mini_cluster.cc:1428] Setting key 326d00524b41918211b93b2a6258679f
WARNING: no policy specified for kudu/127.5.22.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.61@KRBTEST.COM" created.
Sep 28 06:39:56 dist-test-slave-j0qf krb5kdc[27969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577596, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:39:56.457222 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.439444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:35705 (local address 127.0.0.1:44156)
0928 06:39:56.439952 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.440005 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:56.441040 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:56.441801 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:56.441811 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:56.442380 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:56.442782 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:56.442804 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.444625 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.444628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:56.445373 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:56.445383 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.446387 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.446394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:56.446575 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.447181 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:56.447198 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:56.451756 (+ 4558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:56.454115 (+ 2359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:56.454123 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:56.454131 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:56.454445 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:56.454843 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:56.454845 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:56.454846 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:56.454975 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:56.455829 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:56.455834 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:56.455959 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 06:39:56.456094 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 06:39:56.456353 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":55,"thread_start_us":149,"threads_started":1}
I20260928 06:39:56.457227 28053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.439444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44156 (local address 127.5.22.62:35705)
0928 06:39:56.440008 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:56.440026 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:56.440114 (+ 88us) server_negotiation.cc:407] Connection header received
0928 06:39:56.441388 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:56.441392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:56.441583 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:56.441740 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:56.442983 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.444490 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.445506 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.446133 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.446341 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.451933 (+ 5592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:56.451940 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:56.451954 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:56.451984 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:56.453993 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.454557 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.454566 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.454571 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.454731 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.455109 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.455116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.455120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.455701 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:56.455858 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:56.456242 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 06:39:56.456441 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":426,"spinlock_wait_cycles":23296,"thread_start_us":167,"threads_started":1}
I20260928 06:39:56.460079 28050 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.61" port: 36563 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:39:56.460894 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.460644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:36563 (local address 127.0.0.1:36224)
0928 06:39:56.460761 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.460839 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.61:36563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 06:39:56.461086 28001 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.22.61:36563: connect: Connection refused (error 111)
I20260928 06:39:56.462046 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.461852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:36563 (local address 127.0.0.1:36240)
0928 06:39:56.461954 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.461998 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.61:36563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 06:39:56.462188 28050 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.61:36563: Network error: Client connection negotiation failed: client connection to 127.5.22.61:36563: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.61.keytab.
Entry for principal kudu/127.5.22.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.61.keytab.
I20260928 06:39:56.468989 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.61:36563
--webserver_interface=127.5.22.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.61.keytab
--principal=kudu/127.5.22.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:45597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:35705,127.5.22.61:36563,127.5.22.60:45023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:39:56.511451 28050 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.61" port: 36563 } attempt: 1
I20260928 06:39:56.512516 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.512189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:36563 (local address 127.0.0.1:36256)
0928 06:39:56.512359 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.512426 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.61:36563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 06:39:56.513365 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.513143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:36563 (local address 127.0.0.1:36262)
0928 06:39:56.513266 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.513311 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.61:36563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 06:39:56.513530 28050 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.61:36563: Network error: Client connection negotiation failed: client connection to 127.5.22.61:36563: connect: Connection refused (error 111)
I20260928 06:39:56.581714 28050 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.61" port: 36563 } attempt: 2
W20260928 06:39:56.581650 28057 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:39:56.581909 28057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:56.581979 28057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:56.582050 28057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 06:39:56.582682 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.582385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:36563 (local address 127.0.0.1:36268)
0928 06:39:56.582535 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.582607 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.61:36563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 06:39:56.583362 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.583142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:36563 (local address 127.0.0.1:36280)
0928 06:39:56.583250 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.583309 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.61:36563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 06:39:56.583487 28050 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.61:36563: Network error: Client connection negotiation failed: client connection to 127.5.22.61:36563: connect: Connection refused (error 111)
W20260928 06:39:56.586360 28057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:56.586442 28057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:56.586488 28057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:56.586524 28057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:56.586553 28057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:56.591831 28057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:45597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:35705,127.5.22.61:36563,127.5.22.60:45023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.61.keytab
--principal=kudu/127.5.22.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.61:36563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.22.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:56.592882 28057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:56.593894 28057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:39:56.602566 28057 server_base.cc:1034] running on GCE node
W20260928 06:39:56.602527 28065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:56.602523 28062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:56.602726 28063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:56.603267 28057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:56.604334 28057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:56.605540 28057 hybrid_clock.cc:648] HybridClock initialized: now 1790577596605514 us; error 39 us; skew 500 ppm
Sep 28 06:39:56 dist-test-slave-j0qf krb5kdc[27969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577596, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:56.608453 28057 init.cc:377] Logged in from keytab as kudu/127.5.22.61@KRBTEST.COM (short username kudu)
I20260928 06:39:56.609360 28057 webserver.cc:460] Webserver started at http://127.5.22.61:41821/ using document root <none> and password file <none>
I20260928 06:39:56.609912 28057 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:56.609969 28057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:56.610129 28057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:56.612205 28057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-1/data/instance:
uuid: "55e2a61598ff475295ed127eb5abcff1"
format_stamp: "Formatted at 2026-09-28 06:39:56 on dist-test-slave-j0qf"
server_key: "8b5f36cb15ac2887bdb6ea1a2c4edf6e"
server_key_iv: "2f80dc2c73df309769731d082036760d"
server_key_version: "encryptionkey@0"
I20260928 06:39:56.612715 28057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-1/wal/instance:
uuid: "55e2a61598ff475295ed127eb5abcff1"
format_stamp: "Formatted at 2026-09-28 06:39:56 on dist-test-slave-j0qf"
server_key: "8b5f36cb15ac2887bdb6ea1a2c4edf6e"
server_key_iv: "2f80dc2c73df309769731d082036760d"
server_key_version: "encryptionkey@0"
I20260928 06:39:56.616329 28057 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 06:39:56.618770 28072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:56.620016 28057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:39:56.620113 28057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-1/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-1/wal
uuid: "55e2a61598ff475295ed127eb5abcff1"
format_stamp: "Formatted at 2026-09-28 06:39:56 on dist-test-slave-j0qf"
server_key: "8b5f36cb15ac2887bdb6ea1a2c4edf6e"
server_key_iv: "2f80dc2c73df309769731d082036760d"
server_key_version: "encryptionkey@0"
I20260928 06:39:56.620195 28057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:56.641568 28057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:56.644686 28057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:56.644851 28057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:56.652645 28057 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.61:36563
I20260928 06:39:56.653012 28124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.61:36563 every 8 connection(s)
I20260928 06:39:56.653962 28057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-1/data/info.pb
I20260928 06:39:56.655375 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 28057
I20260928 06:39:56.655540 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-1/wal/instance
I20260928 06:39:56.655885 5208 external_mini_cluster.cc:1428] Setting key a1751ce13f8602ad979cc0300664f544
I20260928 06:39:56.657367 28125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:56.661888 28125 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.22.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.22.60@KRBTEST.COM" created.
Sep 28 06:39:56 dist-test-slave-j0qf krb5kdc[27969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577596, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.61@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
I20260928 06:39:56.674656 28050 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.61" port: 36563 } attempt: 3
I20260928 06:39:56.677614 28053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.664151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44172 (local address 127.5.22.62:35705)
0928 06:39:56.664370 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:56.664374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:56.665376 (+ 1002us) server_negotiation.cc:407] Connection header received
0928 06:39:56.666473 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:56.666476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:56.666541 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:56.666650 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:56.668116 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.668757 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.670201 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.670677 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.670808 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.673427 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:56.673433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:56.673436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:56.673468 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:56.675207 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.675806 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.675809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.675811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.675899 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.676317 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.676320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.676322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.676556 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:56.676731 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:56.677321 (+ 590us) server_negotiation.cc:299] Negotiation successful
0928 06:39:56.677441 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":124}
I20260928 06:39:56.678601 28128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.664589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:35705 (local address 127.0.0.1:44172)
0928 06:39:56.665198 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.665261 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:56.666209 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:56.666896 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:56.666910 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:56.667423 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:56.667946 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:56.667960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.668912 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.668919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:56.670058 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:56.670068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.670836 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.670839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:56.670974 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.671611 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:56.671627 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:56.673249 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:56.675332 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:56.675338 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:56.675355 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:56.675708 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:56.676020 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:56.676023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:56.676024 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:56.676205 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:56.676752 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:56.676760 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:56.677023 (+ 263us) client_negotiation.cc:769] Sending connection context
0928 06:39:56.677256 (+ 233us) client_negotiation.cc:240] Negotiation successful
0928 06:39:56.677576 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":179,"threads_started":1}
I20260928 06:39:56.680580 28125 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.61" port: 36563 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:39:56 dist-test-slave-j0qf krb5kdc[27969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577596, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for kudu/127.5.22.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.22.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.60.keytab.
Entry for principal kudu/127.5.22.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.60.keytab.
I20260928 06:39:56.684921 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.675099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:36563 (local address 127.0.0.1:36290)
0928 06:39:56.675255 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.675274 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:56.675361 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:56.676169 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:56.676172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:56.676416 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:56.676549 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:56.676554 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.677690 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.677696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:56.678361 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:56.678377 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.679075 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.679078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:56.679203 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.679661 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:56.679672 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:56.681014 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:56.683162 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:56.683165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:56.683167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:56.683411 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:56.683678 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:56.683681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:56.683689 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:56.683795 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:56.684339 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:56.684342 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:56.684442 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 06:39:56.684578 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 06:39:56.684720 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":18}
I20260928 06:39:56.685745 5208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgGsXow/build/debug/bin/kudu
/tmp/dist-test-taskgGsXow/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.60:45023
--webserver_interface=127.5.22.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.60.keytab
--principal=kudu/127.5.22.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.22.20:45597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.22.62:35705,127.5.22.61:36563,127.5.22.60:45023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:39:56 dist-test-slave-j0qf krb5kdc[27969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577596, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.61@KRBTEST.COM for kudu/127.5.22.61@KRBTEST.COM
I20260928 06:39:56.687685 28130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.675415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36290 (local address 127.5.22.61:36563)
0928 06:39:56.675716 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:56.675725 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:56.675745 (+ 20us) server_negotiation.cc:407] Connection header received
0928 06:39:56.675809 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:56.675814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:56.676036 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:56.676193 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:56.676663 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.677570 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.678483 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.678947 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.679074 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.681151 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:56.681159 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:56.681171 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:56.681195 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:56.683059 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.683509 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.683514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.683515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.683586 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.683900 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.683903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.683905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.684241 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:56.684437 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:56.684753 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 06:39:56.684931 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":104,"threads_started":1}
I20260928 06:39:56.689669 28050 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:39:56.690469 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.690184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:45023 (local address 127.0.0.1:48036)
0928 06:39:56.690344 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.690397 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:45023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 06:39:56.691056 28128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.681104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:36563 (local address 127.0.0.1:36306)
0928 06:39:56.681237 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.681253 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:56.681326 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:56.681907 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:56.681910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:56.682071 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:56.682147 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:56.682152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.682953 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.682956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:56.683695 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:56.683705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.684347 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.684350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:56.684449 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.684860 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:56.684874 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:56.686159 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:56.688886 (+ 2727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:56.688892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:56.688897 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:56.689160 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:56.689625 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:56.689631 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:56.689636 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:56.689740 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:56.690413 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:56.690419 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:56.690534 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 06:39:56.690640 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 06:39:56.690805 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 06:39:56.691210 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.690939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:45023 (local address 127.0.0.1:48048)
0928 06:39:56.691080 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.691134 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:45023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260928 06:39:56.691380 28050 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.60:45023: Network error: Client connection negotiation failed: client connection to 127.5.22.60:45023: connect: Connection refused (error 111)
I20260928 06:39:56.691773 28131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.681262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36306 (local address 127.5.22.61:36563)
0928 06:39:56.681676 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:56.681695 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:56.681709 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:39:56.681745 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:56.681748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:56.681785 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:56.681901 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:56.682266 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.682838 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.683806 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.684154 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.684258 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.686346 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:56.686353 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:56.686359 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:56.686388 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:56.688752 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.689277 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.689280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.689281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.689380 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.689903 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.689916 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.689921 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.690195 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:56.690302 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:56.691350 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0928 06:39:56.691494 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":334,"thread_start_us":133,"threads_started":1}
I20260928 06:39:56.692498 28125 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:39:56.693218 28128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.692981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:45023 (local address 127.0.0.1:48050)
0928 06:39:56.693094 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.693168 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:45023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 06:39:56.693531 28076 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.22.60:45023: connect: Connection refused (error 111)
I20260928 06:39:56.694487 28128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.694242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:45023 (local address 127.0.0.1:48052)
0928 06:39:56.694400 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.694441 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:45023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 06:39:56.694716 28125 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.60:45023: Network error: Client connection negotiation failed: client connection to 127.5.22.60:45023: connect: Connection refused (error 111)
I20260928 06:39:56.722548 28050 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } attempt: 1
I20260928 06:39:56.723711 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.723318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:45023 (local address 127.0.0.1:48054)
0928 06:39:56.723487 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.723563 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:45023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 06:39:56.724625 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.724409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:45023 (local address 127.0.0.1:48068)
0928 06:39:56.724519 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.724568 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:45023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 06:39:56.724913 28050 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.60:45023: Network error: Client connection negotiation failed: client connection to 127.5.22.60:45023: connect: Connection refused (error 111)
I20260928 06:39:56.744009 28125 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } attempt: 1
I20260928 06:39:56.744983 28128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.744665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:45023 (local address 127.0.0.1:48080)
0928 06:39:56.744826 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.744903 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:45023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 06:39:56.745868 28128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.745618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:45023 (local address 127.0.0.1:48094)
0928 06:39:56.745746 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.745806 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:45023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 06:39:56.746059 28125 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.60:45023: Network error: Client connection negotiation failed: client connection to 127.5.22.60:45023: connect: Connection refused (error 111)
I20260928 06:39:56.800097 28050 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } attempt: 2
W20260928 06:39:56.800324 28132 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:39:56.800709 28132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:56.800757 28132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:56.800833 28132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 06:39:56.801028 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.800627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:45023 (local address 127.0.0.1:48102)
0928 06:39:56.800845 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.800938 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:45023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 06:39:56.801733 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.801520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:45023 (local address 127.0.0.1:48114)
0928 06:39:56.801633 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.801685 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:45023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 06:39:56.801880 28050 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.60:45023: Network error: Client connection negotiation failed: client connection to 127.5.22.60:45023: connect: Connection refused (error 111)
W20260928 06:39:56.805328 28132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:56.805382 28132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:56.805401 28132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:56.805466 28132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:56.805500 28132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:56.810950 28132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.22.20:45597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.22.62:35705,127.5.22.61:36563,127.5.22.60:45023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/krb5kdc/kudu_127.5.22.60.keytab
--principal=kudu/127.5.22.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.22.60:45023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.22.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 06:04:39 UTC on ccycloud.quasar-pemmwb.root.comops.site
build id 4676663
I20260928 06:39:56.812006 28132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:56.813088 28132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:39:56.814215 28125 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } attempt: 2
I20260928 06:39:56.815007 28128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.814720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:45023 (local address 127.0.0.1:48120)
0928 06:39:56.814873 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.814935 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:45023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 06:39:56.815781 28128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.815576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:45023 (local address 127.0.0.1:48122)
0928 06:39:56.815683 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.815732 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.22.60:45023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 06:39:56.815966 28125 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.22.60:45023: Network error: Client connection negotiation failed: client connection to 127.5.22.60:45023: connect: Connection refused (error 111)
I20260928 06:39:56.822777 28132 server_base.cc:1034] running on GCE node
W20260928 06:39:56.822705 28141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:56.822810 28138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:56.822722 28139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:56.823521 28132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:56.824747 28132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:56.825941 28132 hybrid_clock.cc:648] HybridClock initialized: now 1790577596825923 us; error 32 us; skew 500 ppm
Sep 28 06:39:56 dist-test-slave-j0qf krb5kdc[27969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577596, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:56.832736 28132 init.cc:377] Logged in from keytab as kudu/127.5.22.60@KRBTEST.COM (short username kudu)
I20260928 06:39:56.833748 28132 webserver.cc:460] Webserver started at http://127.5.22.60:35429/ using document root <none> and password file <none>
I20260928 06:39:56.834479 28132 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:56.834581 28132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:56.834830 28132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:56.836984 28132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-2/data/instance:
uuid: "29b3c5a2355f40e99c2bf66984c38ae3"
format_stamp: "Formatted at 2026-09-28 06:39:56 on dist-test-slave-j0qf"
server_key: "b0ba2b76fd2957966900358885602894"
server_key_iv: "c8608817038a6ef8478a3e2e09a5d6d1"
server_key_version: "encryptionkey@0"
I20260928 06:39:56.837564 28132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-2/wal/instance:
uuid: "29b3c5a2355f40e99c2bf66984c38ae3"
format_stamp: "Formatted at 2026-09-28 06:39:56 on dist-test-slave-j0qf"
server_key: "b0ba2b76fd2957966900358885602894"
server_key_iv: "c8608817038a6ef8478a3e2e09a5d6d1"
server_key_version: "encryptionkey@0"
I20260928 06:39:56.841486 28132 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 06:39:56.844199 28148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:56.845479 28132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 06:39:56.845626 28132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-2/data,/tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-2/wal
uuid: "29b3c5a2355f40e99c2bf66984c38ae3"
format_stamp: "Formatted at 2026-09-28 06:39:56 on dist-test-slave-j0qf"
server_key: "b0ba2b76fd2957966900358885602894"
server_key_iv: "c8608817038a6ef8478a3e2e09a5d6d1"
server_key_version: "encryptionkey@0"
I20260928 06:39:56.845741 28132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:56.863700 28132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:56.866565 28132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:56.866753 28132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:56.873651 28132 rpc_server.cc:307] RPC server started. Bound to: 127.5.22.60:45023
I20260928 06:39:56.873988 28200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.22.60:45023 every 8 connection(s)
I20260928 06:39:56.874626 28132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-2/data/info.pb
I20260928 06:39:56.877336 28201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:56.880906 28201 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:39:56.883838 5208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgGsXow/build/debug/bin/kudu as pid 28132
I20260928 06:39:56.883984 5208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgGsXow/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577308481727-5208-0/minicluster-data/master-2/wal/instance
I20260928 06:39:56.884268 5208 external_mini_cluster.cc:1428] Setting key 9a90015cd7037dbc432a1fa2af4a02be
Sep 28 06:39:56 dist-test-slave-j0qf krb5kdc[27969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577596, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.60@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
Sep 28 06:39:56 dist-test-slave-j0qf krb5kdc[27969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.60@KRBTEST.COM
I20260928 06:39:56.896705 28053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.882680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44186 (local address 127.5.22.62:35705)
0928 06:39:56.882839 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:56.882843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:56.883598 (+ 755us) server_negotiation.cc:407] Connection header received
0928 06:39:56.884664 (+ 1066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:56.884667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:56.884729 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:56.884838 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:56.886408 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.887127 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.888475 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.888701 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.888802 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.892291 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:56.892298 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:56.892302 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:56.892374 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:56.894367 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.894969 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.894973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.894975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.895063 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.895429 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.895433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.895434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.895650 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:56.895762 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:56.896403 (+ 641us) server_negotiation.cc:299] Negotiation successful
0928 06:39:56.896527 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 06:39:56.897521 28203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.883006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.62:35705 (local address 127.0.0.1:44186)
0928 06:39:56.883435 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.883476 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:56.884458 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:56.885014 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:56.885021 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:56.885544 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:56.886181 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:56.886204 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.887241 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.887243 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:56.888353 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:56.888365 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.888847 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.888850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:56.888998 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.889631 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:56.889670 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:56.892061 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:56.894496 (+ 2435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:56.894506 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:56.894537 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:56.894859 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:56.895158 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:56.895162 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:56.895164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:56.895317 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:56.895763 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:56.895771 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:56.896067 (+ 296us) client_negotiation.cc:769] Sending connection context
0928 06:39:56.896347 (+ 280us) client_negotiation.cc:240] Negotiation successful
0928 06:39:56.896678 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":91,"spinlock_wait_cycles":9088,"thread_start_us":108,"threads_started":1}
I20260928 06:39:56.897527 28205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.884945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48136 (local address 127.5.22.60:45023)
0928 06:39:56.885306 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:56.885315 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:56.885335 (+ 20us) server_negotiation.cc:407] Connection header received
0928 06:39:56.885462 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:56.885467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:56.885637 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:56.885758 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:56.886648 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.887695 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.888728 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.889132 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.889210 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.892343 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:56.892352 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:56.892366 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:56.892391 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:56.894853 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.895272 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.895277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.895279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.895382 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.895670 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.895675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.895677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.896053 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:56.896289 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:56.896539 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 06:39:56.896743 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":268,"spinlock_wait_cycles":2432,"thread_start_us":140,"threads_started":1}
I20260928 06:39:56.898936 28201 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.61" port: 36563 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:39:56.901050 28050 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } attempt: 3
Sep 28 06:39:56 dist-test-slave-j0qf krb5kdc[27969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577596, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.60@KRBTEST.COM for kudu/127.5.22.61@KRBTEST.COM
Sep 28 06:39:56 dist-test-slave-j0qf krb5kdc[27969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.62@KRBTEST.COM
Sep 28 06:39:56 dist-test-slave-j0qf krb5kdc[27969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577596, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.62@KRBTEST.COM for kudu/127.5.22.60@KRBTEST.COM
I20260928 06:39:56.907135 28125 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } attempt: 3
I20260928 06:39:56.910382 28053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.899927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44196 (local address 127.5.22.62:35705)
0928 06:39:56.900185 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:56.900188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:56.900198 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:39:56.900227 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:56.900229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:56.900260 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:56.900320 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:56.900942 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.901647 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.902706 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.902932 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.903039 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.905823 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:56.905827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:56.905830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:56.905857 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:56.907857 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.908355 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.908360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.908362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.908454 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.908841 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.908844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.908846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.909091 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:56.909311 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:56.909989 (+ 678us) server_negotiation.cc:299] Negotiation successful
0928 06:39:56.910128 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260928 06:39:56.911535 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.901410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:45023 (local address 127.0.0.1:48138)
0928 06:39:56.901628 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.901642 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:56.901715 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:56.901961 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:56.901964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:56.902141 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:56.902226 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:56.902231 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.903310 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.903314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:56.904744 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:56.904759 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.905221 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.905225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:56.905313 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.905758 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:56.905770 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:56.907684 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:56.909746 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:56.909750 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:56.909752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:56.910031 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:56.910457 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:56.910461 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:56.910464 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:56.910596 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:56.911039 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:56.911045 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:56.911152 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 06:39:56.911264 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 06:39:56.911389 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260928 06:39:56.912075 28205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.901652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48138 (local address 127.5.22.60:45023)
0928 06:39:56.901756 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:56.901759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:56.901770 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:39:56.901798 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:56.901801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:56.901848 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:56.901925 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:56.902391 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.903187 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.904860 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.905110 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.905220 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.907841 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:56.907845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:56.907849 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:56.907874 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:56.909635 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.910162 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.910167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.910169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.910278 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.910721 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.910724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.910726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.910947 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:56.911053 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:56.911803 (+ 750us) server_negotiation.cc:299] Negotiation successful
0928 06:39:56.911913 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":46}
I20260928 06:39:56.912288 28131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.899597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36316 (local address 127.5.22.61:36563)
0928 06:39:56.899721 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:56.899725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:56.899740 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:39:56.899866 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:56.899870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:56.899923 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:56.900029 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:56.900790 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.901499 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.902498 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.902858 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.902933 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.906266 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:56.906269 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:56.906272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:56.906301 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:56.908686 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.909982 (+ 1296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.909988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.909990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.910086 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.910709 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.910712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.910714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.911320 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:56.911426 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:56.911932 (+ 506us) server_negotiation.cc:299] Negotiation successful
0928 06:39:56.912061 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":51}
I20260928 06:39:56.913431 28203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.899462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.61:36563 (local address 127.0.0.1:36316)
0928 06:39:56.899636 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.899651 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:56.899729 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:56.900410 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:56.900412 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:56.900548 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:56.900655 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:56.900661 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.901656 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.901659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:56.902367 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:56.902379 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.903077 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.903080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:56.903269 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.903893 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:56.903904 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:56.906145 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:56.908815 (+ 2670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:56.908820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:56.908825 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:56.909865 (+ 1040us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:56.910201 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:56.910204 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:56.910205 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:56.910337 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:56.911477 (+ 1140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:56.911480 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:56.911573 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 06:39:56.912644 (+ 1071us) client_negotiation.cc:240] Negotiation successful
0928 06:39:56.912790 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":33}
Sep 28 06:39:56 dist-test-slave-j0qf krb5kdc[27969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577596, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.61@KRBTEST.COM for kudu/127.5.22.60@KRBTEST.COM
I20260928 06:39:56.916385 28050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c: Bootstrap starting.
I20260928 06:39:56.918049 28128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.907570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:45023 (local address 127.0.0.1:48150)
0928 06:39:56.907763 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.907779 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:56.907853 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:56.909124 (+ 1271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:56.909127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:56.909308 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:56.909427 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:56.909434 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.911130 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.911134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:56.911969 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:56.911980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.912768 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.912772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:56.912852 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.913321 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:56.913331 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:56.914661 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:56.916449 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:56.916455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:56.916460 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:56.916705 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:56.917045 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:56.917048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:56.917050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:56.917141 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:56.917539 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:56.917542 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:56.917641 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 06:39:56.917749 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 06:39:56.917880 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":24}
Sep 28 06:39:56 dist-test-slave-j0qf krb5kdc[27969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.22.61@KRBTEST.COM
I20260928 06:39:56.919677 28050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:56.920446 28050 log.cc:824] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:56.918073 28206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.907656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48150 (local address 127.5.22.60:45023)
0928 06:39:56.908327 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:56.908700 (+ 373us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:56.908720 (+ 20us) server_negotiation.cc:407] Connection header received
0928 06:39:56.908757 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:56.908762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:56.908840 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:56.909064 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:56.909564 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.910996 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.912237 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.912585 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.913070 (+ 485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.914804 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:56.914807 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:56.914809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:56.914839 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:56.916287 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.916878 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.916880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.916881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.916950 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.917258 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.917263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.917265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.917433 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:56.917547 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:56.917813 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 06:39:56.917920 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":127,"threads_started":1}
I20260928 06:39:56.921558 28201 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:39:56.923050 28050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c: No bootstrap required, opened a new log
I20260928 06:39:56.923170 28130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.911772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36330 (local address 127.5.22.61:36563)
0928 06:39:56.912164 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:56.912168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:56.912180 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:39:56.912505 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:56.912509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:56.912555 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:56.912621 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:56.913732 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.914476 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.915544 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.915863 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.915941 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.919091 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:56.919094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:56.919096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:56.919118 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:56.920714 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.921377 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.921381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.921382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.921470 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.922013 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.922020 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.922024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.922254 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:56.922469 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:56.922850 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 06:39:56.922991 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292}
I20260928 06:39:56.924079 28125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55e2a61598ff475295ed127eb5abcff1: Bootstrap starting.
I20260928 06:39:56.926643 5208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:56.926873 28125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55e2a61598ff475295ed127eb5abcff1: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:56.927634 28050 raft_consensus.cc:348] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90a1a53145347598f2e1e48964bec8c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } } peers { permanent_uuid: "55e2a61598ff475295ed127eb5abcff1" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 36563 } } peers { permanent_uuid: "29b3c5a2355f40e99c2bf66984c38ae3" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } }
I20260928 06:39:56.927901 28050 raft_consensus.cc:729] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f90a1a53145347598f2e1e48964bec8c, State: Initialized, Role: FOLLOWER
I20260928 06:39:56.927966 28024 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f90a1a53145347598f2e1e48964bec8c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44196
I20260928 06:39:56.928256 28050 consensus_queue.cc:260] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90a1a53145347598f2e1e48964bec8c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } } peers { permanent_uuid: "55e2a61598ff475295ed127eb5abcff1" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 36563 } } peers { permanent_uuid: "29b3c5a2355f40e99c2bf66984c38ae3" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } }
I20260928 06:39:56.928776 28125 log.cc:824] T 00000000000000000000000000000000 P 55e2a61598ff475295ed127eb5abcff1: Log is configured to *not* fsync() on all Append() calls
Sep 28 06:39:56 dist-test-slave-j0qf krb5kdc[27969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577596, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.22.60@KRBTEST.COM for kudu/127.5.22.60@KRBTEST.COM
I20260928 06:39:56.929791 28050 sys_catalog.cc:564] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:56.929734 28209 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90a1a53145347598f2e1e48964bec8c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } } peers { permanent_uuid: "55e2a61598ff475295ed127eb5abcff1" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 36563 } } peers { permanent_uuid: "29b3c5a2355f40e99c2bf66984c38ae3" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } } }
I20260928 06:39:56.929914 28209 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:39:56.930028 28024 raft_consensus.cc:482] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 06:39:56.930167 28024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:56.931427 28125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55e2a61598ff475295ed127eb5abcff1: No bootstrap required, opened a new log
I20260928 06:39:56.932101 28024 raft_consensus.cc:504] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90a1a53145347598f2e1e48964bec8c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } } peers { permanent_uuid: "55e2a61598ff475295ed127eb5abcff1" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 36563 } } peers { permanent_uuid: "29b3c5a2355f40e99c2bf66984c38ae3" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } }
I20260928 06:39:56.933542 28203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.922144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.22.60:45023 (local address 127.0.0.1:48158)
0928 06:39:56.922578 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:56.922592 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:56.922661 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:56.924501 (+ 1840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:56.924504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:56.924753 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:56.924865 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:56.924876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.925835 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.925837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:56.926638 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:56.926649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.927287 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.927292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:56.927363 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.927868 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:56.927882 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:56.929423 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:56.931904 (+ 2481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:56.931907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:56.931909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:56.932171 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:56.932544 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:56.932547 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:56.932548 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:56.932628 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:56.933029 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:56.933043 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:56.933130 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 06:39:56.933225 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 06:39:56.933361 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":164,"threads_started":1}
I20260928 06:39:56.934192 28206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.922534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48158 (local address 127.5.22.60:45023)
0928 06:39:56.924291 (+ 1757us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:56.924295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:56.924309 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:39:56.924341 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:56.924344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:56.924382 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:56.924469 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:56.924984 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.925723 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.926756 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.927067 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.927171 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.929589 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:56.929593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:56.929595 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:56.929620 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:56.931762 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.932361 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.932364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.932366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.932448 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.932714 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.932717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.932719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.932911 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:56.932999 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:56.933937 (+ 938us) server_negotiation.cc:299] Negotiation successful
0928 06:39:56.934049 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1684}
I20260928 06:39:56.933082 28024 leader_election.cc:288] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c [CANDIDATE]: Term 1 election: Requested vote from peers 55e2a61598ff475295ed127eb5abcff1 (127.5.22.61:36563), 29b3c5a2355f40e99c2bf66984c38ae3 (127.5.22.60:45023)
W20260928 06:39:56.935760 28221 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:39:56.935832 28221 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:39:56.937395 28201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b3c5a2355f40e99c2bf66984c38ae3: Bootstrap starting.
I20260928 06:39:56.936023 27982 master_runner.cc:186] Error getting master registration for 127.5.22.61:36563: OK, instance_id { permanent_uuid: "55e2a61598ff475295ed127eb5abcff1" instance_seqno: 1790577596650486 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:39:56.940099 27982 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:39:56.940090 28201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29b3c5a2355f40e99c2bf66984c38ae3: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:56.940464 28125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55e2a61598ff475295ed127eb5abcff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90a1a53145347598f2e1e48964bec8c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } } peers { permanent_uuid: "55e2a61598ff475295ed127eb5abcff1" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 36563 } } peers { permanent_uuid: "29b3c5a2355f40e99c2bf66984c38ae3" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } }
I20260928 06:39:56.940681 28125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55e2a61598ff475295ed127eb5abcff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e2a61598ff475295ed127eb5abcff1, State: Initialized, Role: FOLLOWER
I20260928 06:39:56.940936 28201 log.cc:824] T 00000000000000000000000000000000 P 29b3c5a2355f40e99c2bf66984c38ae3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:56.941006 28125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55e2a61598ff475295ed127eb5abcff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90a1a53145347598f2e1e48964bec8c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } } peers { permanent_uuid: "55e2a61598ff475295ed127eb5abcff1" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 36563 } } peers { permanent_uuid: "29b3c5a2355f40e99c2bf66984c38ae3" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } }
I20260928 06:39:56.941972 28230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55e2a61598ff475295ed127eb5abcff1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90a1a53145347598f2e1e48964bec8c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } } peers { permanent_uuid: "55e2a61598ff475295ed127eb5abcff1" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 36563 } } peers { permanent_uuid: "29b3c5a2355f40e99c2bf66984c38ae3" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } } }
I20260928 06:39:56.942131 28230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55e2a61598ff475295ed127eb5abcff1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:39:56.942370 28125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55e2a61598ff475295ed127eb5abcff1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:56.942436 28099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f90a1a53145347598f2e1e48964bec8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "55e2a61598ff475295ed127eb5abcff1"
I20260928 06:39:56.942628 28099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55e2a61598ff475295ed127eb5abcff1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:56.944051 28099 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 55e2a61598ff475295ed127eb5abcff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f90a1a53145347598f2e1e48964bec8c in term 1.
I20260928 06:39:56.944511 28001 leader_election.cc:302] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55e2a61598ff475295ed127eb5abcff1, f90a1a53145347598f2e1e48964bec8c; no voters:
I20260928 06:39:56.945019 28209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:56.945427 28209 raft_consensus.cc:686] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c [term 1 LEADER]: Becoming Leader. State: Replica: f90a1a53145347598f2e1e48964bec8c, State: Running, Role: LEADER
I20260928 06:39:56.946092 28209 consensus_queue.cc:237] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90a1a53145347598f2e1e48964bec8c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } } peers { permanent_uuid: "55e2a61598ff475295ed127eb5abcff1" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 36563 } } peers { permanent_uuid: "29b3c5a2355f40e99c2bf66984c38ae3" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } }
W20260928 06:39:56.948678 28242 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 55e2a61598ff475295ed127eb5abcff1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:39:56.948755 28242 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:39:56.950842 28201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b3c5a2355f40e99c2bf66984c38ae3: No bootstrap required, opened a new log
I20260928 06:39:56.952628 28053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.938672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44202 (local address 127.5.22.62:35705)
0928 06:39:56.938801 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:56.938806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:56.942599 (+ 3793us) server_negotiation.cc:407] Connection header received
0928 06:39:56.942818 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:56.942821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:56.942871 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:56.942921 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:56.943977 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.944821 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.946155 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.946511 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.946588 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.948543 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:56.948550 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:56.948554 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:56.948579 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:56.950531 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.951125 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.951130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.951133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.951241 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.951608 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.951618 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.951619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.951841 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:56.951954 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:56.952361 (+ 407us) server_negotiation.cc:299] Negotiation successful
0928 06:39:56.952477 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:39:56.955415 28057 master_runner.cc:186] Error getting master registration for 127.5.22.60:45023: OK, instance_id { permanent_uuid: "29b3c5a2355f40e99c2bf66984c38ae3" instance_seqno: 1790577596871752 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:39:56.958801 28057 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:39:56.959049 28209 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f90a1a53145347598f2e1e48964bec8c. Latest consensus state: current_term: 1 leader_uuid: "f90a1a53145347598f2e1e48964bec8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90a1a53145347598f2e1e48964bec8c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } } peers { permanent_uuid: "55e2a61598ff475295ed127eb5abcff1" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 36563 } } peers { permanent_uuid: "29b3c5a2355f40e99c2bf66984c38ae3" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } } }
I20260928 06:39:56.959333 28209 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:56.962091 28130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.939128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36332 (local address 127.5.22.61:36563)
0928 06:39:56.939236 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:56.939241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:56.953258 (+ 14017us) server_negotiation.cc:407] Connection header received
0928 06:39:56.953435 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:56.953439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:56.953484 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:56.953532 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:56.954431 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.955214 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.956315 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.956665 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.956746 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.958440 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:56.958445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:56.958447 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:56.958473 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:56.960158 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.960649 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.960652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.960654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.960749 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.961100 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.961106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.961110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.961316 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:56.961438 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:56.961815 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 06:39:56.961930 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:39:56.963142 28201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29b3c5a2355f40e99c2bf66984c38ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90a1a53145347598f2e1e48964bec8c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } } peers { permanent_uuid: "55e2a61598ff475295ed127eb5abcff1" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 36563 } } peers { permanent_uuid: "29b3c5a2355f40e99c2bf66984c38ae3" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } }
I20260928 06:39:56.963425 28201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29b3c5a2355f40e99c2bf66984c38ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b3c5a2355f40e99c2bf66984c38ae3, State: Initialized, Role: FOLLOWER
I20260928 06:39:56.963804 28201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29b3c5a2355f40e99c2bf66984c38ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90a1a53145347598f2e1e48964bec8c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } } peers { permanent_uuid: "55e2a61598ff475295ed127eb5abcff1" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 36563 } } peers { permanent_uuid: "29b3c5a2355f40e99c2bf66984c38ae3" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } }
I20260928 06:39:56.964322 28247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b3c5a2355f40e99c2bf66984c38ae3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90a1a53145347598f2e1e48964bec8c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } } peers { permanent_uuid: "55e2a61598ff475295ed127eb5abcff1" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 36563 } } peers { permanent_uuid: "29b3c5a2355f40e99c2bf66984c38ae3" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } } }
I20260928 06:39:56.964457 28247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b3c5a2355f40e99c2bf66984c38ae3 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:39:56.965276 28201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29b3c5a2355f40e99c2bf66984c38ae3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:56.965754 28175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f90a1a53145347598f2e1e48964bec8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "29b3c5a2355f40e99c2bf66984c38ae3"
I20260928 06:39:56.965965 28175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29b3c5a2355f40e99c2bf66984c38ae3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:56.967440 28175 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 29b3c5a2355f40e99c2bf66984c38ae3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f90a1a53145347598f2e1e48964bec8c in term 1.
I20260928 06:39:56.968665 28246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:56.970793 28246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260928 06:39:56.973325 28258 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 29b3c5a2355f40e99c2bf66984c38ae3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:39:56.973448 28258 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:39:56.974756 28206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.939401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48170 (local address 127.5.22.60:45023)
0928 06:39:56.939532 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:56.939536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:56.960061 (+ 20525us) server_negotiation.cc:407] Connection header received
0928 06:39:56.960360 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:56.960364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:56.960417 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:56.960471 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:56.962078 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.962824 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.965182 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.965491 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.965580 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.970971 (+ 5391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:56.970975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:56.970978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:56.971007 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:56.972633 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.973140 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.973150 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.973151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.973234 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:56.973698 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:56.973702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:56.973704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:56.973913 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:56.974054 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:56.974479 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 06:39:56.974602 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 06:39:56.983032 28175 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 29b3c5a2355f40e99c2bf66984c38ae3 [term 1 FOLLOWER]: Refusing update from remote peer f90a1a53145347598f2e1e48964bec8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:56.983229 28099 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 55e2a61598ff475295ed127eb5abcff1 [term 1 FOLLOWER]: Refusing update from remote peer f90a1a53145347598f2e1e48964bec8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:56.983850 28243 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e2a61598ff475295ed127eb5abcff1" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 36563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:56.984304 28243 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b3c5a2355f40e99c2bf66984c38ae3" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:56.987805 28247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b3c5a2355f40e99c2bf66984c38ae3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f90a1a53145347598f2e1e48964bec8c. Latest consensus state: current_term: 1 leader_uuid: "f90a1a53145347598f2e1e48964bec8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90a1a53145347598f2e1e48964bec8c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } } peers { permanent_uuid: "55e2a61598ff475295ed127eb5abcff1" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 36563 } } peers { permanent_uuid: "29b3c5a2355f40e99c2bf66984c38ae3" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } } }
I20260928 06:39:56.987823 28230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55e2a61598ff475295ed127eb5abcff1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f90a1a53145347598f2e1e48964bec8c. Latest consensus state: current_term: 1 leader_uuid: "f90a1a53145347598f2e1e48964bec8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90a1a53145347598f2e1e48964bec8c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } } peers { permanent_uuid: "55e2a61598ff475295ed127eb5abcff1" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 36563 } } peers { permanent_uuid: "29b3c5a2355f40e99c2bf66984c38ae3" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } } }
I20260928 06:39:56.987921 28247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b3c5a2355f40e99c2bf66984c38ae3 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:39:56.987936 28230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55e2a61598ff475295ed127eb5abcff1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:39:56.990891 28209 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f90a1a53145347598f2e1e48964bec8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90a1a53145347598f2e1e48964bec8c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } } peers { permanent_uuid: "55e2a61598ff475295ed127eb5abcff1" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 36563 } } peers { permanent_uuid: "29b3c5a2355f40e99c2bf66984c38ae3" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } } }
I20260928 06:39:56.991029 28209 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:56.992224 28247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b3c5a2355f40e99c2bf66984c38ae3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f90a1a53145347598f2e1e48964bec8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90a1a53145347598f2e1e48964bec8c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } } peers { permanent_uuid: "55e2a61598ff475295ed127eb5abcff1" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 36563 } } peers { permanent_uuid: "29b3c5a2355f40e99c2bf66984c38ae3" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } } }
I20260928 06:39:56.992348 28247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b3c5a2355f40e99c2bf66984c38ae3 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:39:56.992645 28246 catalog_manager.cc:1347] Generated new cluster ID: 54ffc8fdb676496d983e0c4000ee96af
I20260928 06:39:56.992743 28246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:56.992960 28260 mvcc.cc:205] Tried to move back new op lower bound from 7334205837236039680 to 7334205837097857024. Current Snapshot: MvccSnapshot[applied={T|T < 7334205837236039680 or (T in {7334205837236039680})}]
I20260928 06:39:56.995083 28261 mvcc.cc:205] Tried to move back new op lower bound from 7334205837236039680 to 7334205837097857024. Current Snapshot: MvccSnapshot[applied={T|T < 7334205837236039680}]
I20260928 06:39:56.999266 28243 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f90a1a53145347598f2e1e48964bec8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90a1a53145347598f2e1e48964bec8c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } } peers { permanent_uuid: "55e2a61598ff475295ed127eb5abcff1" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 36563 } } peers { permanent_uuid: "29b3c5a2355f40e99c2bf66984c38ae3" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } } }
I20260928 06:39:56.999414 28243 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:57.000761 28230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55e2a61598ff475295ed127eb5abcff1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f90a1a53145347598f2e1e48964bec8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90a1a53145347598f2e1e48964bec8c" member_type: VOTER last_known_addr { host: "127.5.22.62" port: 35705 } } peers { permanent_uuid: "55e2a61598ff475295ed127eb5abcff1" member_type: VOTER last_known_addr { host: "127.5.22.61" port: 36563 } } peers { permanent_uuid: "29b3c5a2355f40e99c2bf66984c38ae3" member_type: VOTER last_known_addr { host: "127.5.22.60" port: 45023 } } }
I20260928 06:39:57.000895 28230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55e2a61598ff475295ed127eb5abcff1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:39:57.001765 28262 mvcc.cc:205] Tried to move back new op lower bound from 7334205837236039680 to 7334205837097857024. Current Snapshot: MvccSnapshot[applied={T|T < 7334205837236039680}]
I20260928 06:39:57.056260 28246 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:57.057953 28246 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:57.074883 28246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f90a1a53145347598f2e1e48964bec8c: Generated new TSK 0
I20260928 06:39:57.075953 28246 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:39:57.141345 28053 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:57.134949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44214 (local address 127.5.22.62:35705)
0928 06:39:57.139804 (+ 4855us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:57.139810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:57.140100 (+ 290us) server_negotiation.cc:407] Connection header received
0928 06:39:57.140337 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:57.140341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:57.140395 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:57.140453 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:57.141195 (+ 742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44214: BlockingRecv error: recv got EOF from 127.0.0.1:44214 (error 108)
Metrics: {"server-negotiator.queue_time_us":4723}
W20260928 06:39:57.143047 28206 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:57.136377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48172 (local address 127.5.22.60:45023)
0928 06:39:57.136497 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:57.136502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:57.141954 (+ 5452us) server_negotiation.cc:407] Connection header received
0928 06:39:57.142074 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:57.142077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:57.142135 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:57.142237 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:57.142899 (+ 662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48172: BlockingRecv error: recv got EOF from 127.0.0.1:48172 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 06:39:57.144385 28130 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:57.135699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36340 (local address 127.5.22.61:36563)
0928 06:39:57.135832 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:57.135838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:57.141787 (+ 5949us) server_negotiation.cc:407] Connection header received
0928 06:39:57.143531 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:57.143535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:57.143593 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:57.143657 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:57.144230 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36340: BlockingRecv error: recv got EOF from 127.0.0.1:36340 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 06:39:57.149610 28130 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:57.147813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36354 (local address 127.5.22.61:36563)
0928 06:39:57.147927 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:57.147932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:57.148558 (+ 626us) server_negotiation.cc:407] Connection header received
0928 06:39:57.148806 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:57.148812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:57.148893 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:57.148952 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:57.149503 (+ 551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36354: BlockingRecv error: recv got EOF from 127.0.0.1:36354 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 06:39:57.154855 28130 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:57.153181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36356 (local address 127.5.22.61:36563)
0928 06:39:57.153291 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:57.153295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:57.153868 (+ 573us) server_negotiation.cc:407] Connection header received
0928 06:39:57.154065 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:57.154068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:57.154127 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:57.154185 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:57.154750 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36356: BlockingRecv error: recv got EOF from 127.0.0.1:36356 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 06:39:57.160693 28130 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:57.159001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36368 (local address 127.5.22.61:36563)
0928 06:39:57.159106 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:57.159111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:57.159700 (+ 589us) server_negotiation.cc:407] Connection header received
0928 06:39:57.159891 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:57.159894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:57.159945 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:57.159997 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:57.160569 (+ 572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36368: BlockingRecv error: recv got EOF from 127.0.0.1:36368 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 06:39:57.169057 28130 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:57.166991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36382 (local address 127.5.22.61:36563)
0928 06:39:57.167135 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:57.167140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:57.167923 (+ 783us) server_negotiation.cc:407] Connection header received
0928 06:39:57.168139 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:57.168142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:57.168198 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:57.168250 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:57.168884 (+ 634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36382: BlockingRecv error: recv got EOF from 127.0.0.1:36382 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 06:39:57.180708 28130 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:57.179404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36392 (local address 127.5.22.61:36563)
0928 06:39:57.179522 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:57.179527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:57.179778 (+ 251us) server_negotiation.cc:407] Connection header received
0928 06:39:57.179918 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:57.179922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:57.179980 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:57.180081 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:57.180589 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36392: BlockingRecv error: recv got EOF from 127.0.0.1:36392 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 06:39:57.200260 28130 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:57.198971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36398 (local address 127.5.22.61:36563)
0928 06:39:57.199292 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:57.199297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:57.199317 (+ 20us) server_negotiation.cc:407] Connection header received
0928 06:39:57.199480 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:57.199483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:57.199537 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:57.199630 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:57.200135 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36398: BlockingRecv error: recv got EOF from 127.0.0.1:36398 (error 108)
Metrics: {"server-negotiator.queue_time_us":240}
W20260928 06:39:57.236162 28130 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:57.235131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36400 (local address 127.5.22.61:36563)
0928 06:39:57.235273 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:57.235277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:57.235367 (+ 90us) server_negotiation.cc:407] Connection header received
0928 06:39:57.235514 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:57.235516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:57.235565 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:57.235633 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:57.236064 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36400: BlockingRecv error: recv got EOF from 127.0.0.1:36400 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 06:39:57.303673 28130 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:57.302405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36406 (local address 127.5.22.61:36563)
0928 06:39:57.302554 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:57.302560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:57.302750 (+ 190us) server_negotiation.cc:407] Connection header received
0928 06:39:57.302941 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:57.302944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:57.302992 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:57.303083 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:57.303570 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36406: BlockingRecv error: recv got EOF from 127.0.0.1:36406 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 06:39:57.435406 28130 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:57.433999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36410 (local address 127.5.22.61:36563)
0928 06:39:57.434135 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:57.434139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:57.434536 (+ 397us) server_negotiation.cc:407] Connection header received
0928 06:39:57.434639 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:57.434652 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:57.434718 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:57.434833 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:57.435293 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36410: BlockingRecv error: recv got EOF from 127.0.0.1:36410 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 06:39:57.695160 28130 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:57.693874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36422 (local address 127.5.22.61:36563)
0928 06:39:57.694033 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:57.694038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:57.694148 (+ 110us) server_negotiation.cc:407] Connection header received
0928 06:39:57.694359 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:57.694362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:57.694425 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:57.694522 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:57.695010 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36422: BlockingRecv error: recv got EOF from 127.0.0.1:36422 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 06:39:57.950196 28242 catalog_manager.cc:1259] Loaded cluster ID: 54ffc8fdb676496d983e0c4000ee96af
I20260928 06:39:57.950280 28242 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 55e2a61598ff475295ed127eb5abcff1: loading cluster ID for follower catalog manager: success
I20260928 06:39:57.952005 28242 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 55e2a61598ff475295ed127eb5abcff1: acquiring CA information for follower catalog manager: success
I20260928 06:39:57.953169 28242 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 55e2a61598ff475295ed127eb5abcff1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:39:57.974846 28258 catalog_manager.cc:1259] Loaded cluster ID: 54ffc8fdb676496d983e0c4000ee96af
I20260928 06:39:57.974932 28258 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 29b3c5a2355f40e99c2bf66984c38ae3: loading cluster ID for follower catalog manager: success
I20260928 06:39:57.976675 28258 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 29b3c5a2355f40e99c2bf66984c38ae3: acquiring CA information for follower catalog manager: success
I20260928 06:39:57.977825 28258 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 29b3c5a2355f40e99c2bf66984c38ae3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:39:58.213757 28333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.209496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36438 (local address 127.5.22.61:36563)
0928 06:39:58.209795 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.209803 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.209823 (+ 20us) server_negotiation.cc:407] Connection header received
0928 06:39:58.210013 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.210022 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.210146 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.210284 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:58.210615 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.211181 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.212191 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.212429 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.212540 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:58.212884 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:58.213330 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:58.213486 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:58.213522 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 06:39:58.213580 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":82,"threads_started":1}
I20260928 06:39:58.221081 28339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.217218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48184 (local address 127.5.22.60:45023)
0928 06:39:58.217438 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.217442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.217466 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:39:58.217570 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.217572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.217695 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.217793 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:58.218033 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.218702 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.219548 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.219777 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.219907 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:58.220207 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:58.220645 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:58.220749 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:58.220824 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 06:39:58.220875 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":58,"threads_started":1}
I20260928 06:39:58.228888 28333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.224804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36454 (local address 127.5.22.61:36563)
0928 06:39:58.224955 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.224962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.224979 (+ 17us) server_negotiation.cc:407] Connection header received
0928 06:39:58.225262 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.225265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.225330 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.225402 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:58.225804 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.226389 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.227637 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.227988 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.228096 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:58.228371 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:58.228457 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:58.228530 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:58.228664 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 06:39:58.228740 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 06:39:58.229858 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.224478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.5.22.62:35705)
0928 06:39:58.224744 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.224748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.224764 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:39:58.225027 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.225034 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.225232 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.225332 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:58.226408 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.227124 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.228257 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.228637 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.228740 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:58.229055 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:58.229442 (+ 387us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:58.229536 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:58.229633 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 06:39:58.229704 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":76,"threads_started":1}
I20260928 06:39:58.230628 28339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.225129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48198 (local address 127.5.22.60:45023)
0928 06:39:58.225269 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.225274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.225665 (+ 391us) server_negotiation.cc:407] Connection header received
0928 06:39:58.226444 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.226449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.226513 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.226596 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:58.226884 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.227657 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.229102 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.229403 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.229516 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:58.229821 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:58.229939 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:58.230304 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:58.230424 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 06:39:58.230478 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 06:39:58.233379 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 27982
I20260928 06:39:58.240506 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 28057
I20260928 06:39:58.247782 5208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgGsXow/build/debug/bin/kudu with pid 28132
2026-09-28T06:39:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3195 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6346 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (289759 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST